Recommendation for Debug / review
Debugging & Code Review
Our top recommendation for Debugging & Code Review, based on the public evidence we track, is OpenAI: GPT-5.6 Sol.[1][2][3][4] Use for autonomous PR review pipelines where the model must both implement fixes and self-review, as seen in Codex CLI workflows with gpt-5.6-sol pinned for review jobs. Watch out: Watch for review sessions that consume real model spend but fail to write rollout files to ~/.codex/sessions/, making spend invisible to local audit tools. Anthropic: Claude Fable 5 is the next-ranked alternative. Use as the dedicated security and code review lane when registry routing supports tiered assignment, per the recommended upstream policy for plan-review/code-review/advise/security.
About this recommendation
- Updated
- Sep 4, 2026
- Evidence through
- Sep 4, 2026
- Sources
- 28
- Revision
- v61
Decision audit
Why this result
Inspect the inputs and the computed order behind the recommendation.
Models screened
22
live candidates
Evaluation feeds
6
task-weighted
Winner coverage
86%
intended feed weight
Largest provider share
6 of 11
Anthropic
Sources evaluated
The task sets these weights before any model is scored.
| Evaluation feed | Weight | Winner result | Field measured |
|---|---|---|---|
| SWE-rebench | 30% | #4 | 16/22 |
| Aider Polyglot | 20% | not measured | 4/22 |
| LiveBench Coding | 15% | #3 | 15/22 |
| LMArena WebDev | 15% | #7 | 16/22 |
| OpenRouter usage | 10% | 97/100 | 22/22 |
| SWE-bench Verified | 10% | not measured | 5/22 |
Provider concentration
Each exact model is scored separately; provider identity is not a ranking input.
- Anthropic6 models
- OpenAI3 models
- Google1 model
- Z.ai1 model
Decision table
Every published model is shown in computed order. Practitioner sources are distinct community threads, not the citations repeated in the prose below.
| Rank | Model | Relative score | Coverage | Practitioner evidence | Strongest measured reason |
|---|---|---|---|---|---|
| 01 | GPT-5.6 SolOpenAI | 80 | 86% | 12 threads · 11 families · 2 cautions | #3 LiveBench Coding · #4 SWE-rebench |
| 02 | Claude Fable 5Anthropic | 80 | 86% | 5 threads · 4 families · 0 cautions | #1 SWE-rebench · #2 LiveBench Coding |
| 03 | GLM 5.2Z.ai | 72 | 86% | 13 threads · 6 families · 7 cautions | #9 SWE-rebench · #12 LiveBench Coding |
| 04 | Claude Opus 4.8Anthropic | 72 | 86% | 11 threads · 4 families · 4 cautions | #7 LiveBench Coding · #11 SWE-rebench |
| 05 | Claude Sonnet 5Anthropic | 69 | 86% | 2 threads · 2 families · 0 cautions | #10 SWE-rebench · #11 LiveBench Coding |
| 06 | Claude Sonnet 4.6Anthropic | 66 | 86% | 2 threads · 2 families · 2 cautions | #13 SWE-rebench · #14 LiveBench Coding |
| 07 | Claude Opus 4.6Anthropic | 64 | 86% | 1 threads · 1 families · 1 cautions | #16 SWE-rebench · #17 LMArena WebDev |
| 08 | Claude Opus 4.5Anthropic | 64 | 90% | 1 threads · 1 families · 1 cautions | #1 SWE-bench Verified · #12 LiveBench Coding |
| 09 | o3OpenAI | 46 | 68% | 1 threads · 1 families · 0 cautions | #3 Aider Polyglot · #14 SWE-bench Verified |
| 10 | Gemini 2.5 Pro Preview 06-05Google | 27 | 28% | no linked practitioner threads | #2 Aider Polyglot · #6 SWE-bench Verified |
| 11 | o3 ProOpenAI | 26 | 23% | no linked practitioner threads | #1 Aider Polyglot |
Relative score combines normalized benchmark quality and signal coverage; independent practitioner evidence and freshness are bounded tie-breakers. It is an ordering score, not an absolute quality percentage. The writing model receives this order and cannot change it.
GPT-5.6 Sol is deployed as the primary implementation and review lane in production code review workflows, with explicit registry routing for autonomous review fixes and adversarial design consultation.
Best when: Use for autonomous PR review pipelines where the model must both implement fixes and self-review, as seen in Codex CLI workflows with `gpt-5.6-sol` pinned for review jobs.
Tips
- Use for autonomous PR review pipelines where the model must both implement fixes and self-review, as seen in Codex CLI workflows with `gpt-5.6-sol` pinned for review jobs.
- Deploy for adversarial design review where a third independent opinion is needed, specifically in cross-model consult workflows alongside GLM 5.2.
Watch out for
- Watch for review sessions that consume real model spend but fail to write rollout files to `~/.codex/sessions/`, making spend invisible to local audit tools.
- Note that subagent code review may flag working code as 98% useless, so implement pre-commit subagent review before allowing commits in autonomous workflows.
Claude Fable 5 is explicitly designated as the
ultratier for plan-review, code-review, advise, and security tasks in production model taxonomies, with strong benchmark performance on code generation and completion.Best when: Use as the dedicated security and code review lane when registry routing supports tiered assignment, per the recommended upstream policy for `plan-review/code-review/advise/security`.
Tips
- Use as the dedicated security and code review lane when registry routing supports tiered assignment, per the recommended upstream policy for `plan-review/code-review/advise/security`.
- Deploy for automated security reviews of untrusted legacy data parsing code, where the model successfully identified vulnerabilities and opened fix PRs in Pomme (resource forks, PICT, sound).
- Leverage for cost-optimized Dependabot PR review routing, where server-side logic can downgrade patch/minor reviews to cheaper models while keeping majors on Fable 5.
Watch out for
- Verify that workflow subagents actually spawn on the current session model after mid-session `/model` switches, as agents may inherit the original startup model instead.
GLM 5.2 is qualified as an open-weight alternative for cross-model design consultation and adversarial review, with verified deployment through OpenRouter for consumer PR review workflows.
Best when: Use as the third independent opinion in design review workflows, where design flaws survive test-centric gates and need explicit cross-model validation.
Tips
- Use as the third independent opinion in design review workflows, where design flaws survive test-centric gates and need explicit cross-model validation.
- Deploy for cost-sensitive PR review automation where OpenRouter routing to `z-ai/glm-5.2` succeeded across five consumer repositories with posted review comments.
Watch out for
- Watch for silent stalls where review subagents hit 900-second timeouts with no tokens flowing and no error output, requiring external SIGTERM.
- Note significantly lower performance on Frontier-Bench (4.59%) compared to top-tier models, suggesting limited capability on diverse, difficult agent work.
Claude Opus 4.8 serves as the
heavytier for roadmap, plan, and supervise tasks, with demonstrated capability in finding subtle compile/runtime issues and security vulnerabilities that other models miss.Best when: Use as the second-seat reviewer in two-seat certification gates, where it found a P1 issue that six GPT-5.6 Sol legs missed, exposing defects in launcher specs.
Tips
- Use as the second-seat reviewer in two-seat certification gates, where it found a P1 issue that six GPT-5.6 Sol legs missed, exposing defects in launcher specs.
- Deploy for deep vulnerability research with documented Ruby classes and methods, where it identified polymorphic comparison bugs and lazy Enumerator typing issues that were manually verified.
- Leverage for Minecraft mod debugging with crash reports and JAR analysis, successfully identifying and fixing packet serialization bugs in Replay Mod.
Watch out for
- Watch for guardrail false positives where the model requests cybersecurity exemptions for routine debugging tasks like pure virtual call bugs.
Claude Sonnet 5 is designated as the
regulartier for execute and repair tasks with escalation paths to Opus 5, showing moderate performance on SWE-rebench and LiveBench Coding.Best when: Use as the default implementation tier with explicit escalation to Opus 5 after repeated verification failure, per the recommended five-model advisor taxonomy.
Tips
- Use as the default implementation tier with explicit escalation to Opus 5 after repeated verification failure, per the recommended five-model advisor taxonomy.
- Deploy for manual premium review triggers in hybrid PR-Agent workflows, where Sonnet 5 is specified as the upgrade path from Haiku 4.5 for complex changes.
Watch out for
- Expect lower Frontier-Bench performance (14.59%) compared to Fable 5 and Opus tiers, indicating reduced capability on the most difficult agent tasks.
Claude Sonnet 4.6 demonstrated faster bug resolution than Qwen3.6-27B on a specific regression, with LiveBench Coding score of 79.27%.
Best when: Use when faster time-to-fix matters more than raw capability, as Sonnet 4.6 resolved a bug in under 10 minutes where Qwen3.6-27B failed overnight.
Tips
- Use when faster time-to-fix matters more than raw capability, as Sonnet 4.6 resolved a bug in under 10 minutes where Qwen3.6-27B failed overnight.
Watch out for
- Note lower LiveBench Coding ranking (#15 of 51) compared to newer Sonnet 5 and Fable 5 tiers.
Claude Opus 4.6 scores 78.18% on LiveBench Coding and can perform source vulnerability analysis and pentesting without jailbreaking given proper harnesses.
Best when: Use for vulnerability research with source code where guardrail constraints are acceptable, as it finds vulns and pentests without source given proper harnesses.
Tips
- Use for vulnerability research with source code where guardrail constraints are acceptable, as it finds vulns and pentests without source given proper harnesses.
Watch out for
- Expect lower LiveBench Coding performance (#23 of 51) compared to Opus 4.8 and Fable 5 tiers.
Claude Opus 4.5 scores 79.65% on LiveBench Coding and has been used in GitHub Copilot for bug finding harnesses with Gemini 3 Flash.
Best when: Use in Copilot-integrated workflows where combined Opus 4.5 and Gemini 3 Flash harnesses can find bugs in accessible targets like Firefox.
Tips
- Use in Copilot-integrated workflows where combined Opus 4.5 and Gemini 3 Flash harnesses can find bugs in accessible targets like Firefox.
Watch out for
- Note mid-tier LiveBench Coding ranking (#14 of 51) with newer Opus versions available.
o3 scores 81.3% on the Aider polyglot coding benchmark, ranking fourth for editing real code across many languages.
Best when: Consider only after reviewing the cited caution.
Watch out for
- Watch for reasoning content invisibility in evaluation scorers, where `ContentReasoning` items are dropped and jailbreaks in reasoning content may score as empty.
Google: Gemini 2.5 Pro Preview 06-05 scores 83.1% on the Aider polyglot coding benchmark (#3 of 28), which tests editing real code across many languages.
Best when: Consider only after reviewing the cited caution.
OpenAI: o3 Pro scores 84.9% on the Aider polyglot coding benchmark (#2 of 28), which tests editing real code across many languages.
Best when: Consider only after reviewing the cited caution.
Frequently asked
- What is the top-ranked model for Debugging & Code Review?
- OpenAI: GPT-5.6 Sol ranks first in the current evidence-weighted comparison. Use for autonomous PR review pipelines where the model must both implement fixes and self-review, as seen in Codex CLI workflows with `gpt-5.6-sol` pinned for review jobs.[1][2]
- What should I watch out for with OpenAI: GPT-5.6 Sol?
- Watch for review sessions that consume real model spend but fail to write rollout files to `~/.codex/sessions/`, making spend invisible to local audit tools.[3]
- What is an alternative to OpenAI: GPT-5.6 Sol?
- Anthropic: Claude Fable 5 is the next-ranked option. Use as the dedicated security and code review lane when registry routing supports tiered assignment, per the recommended upstream policy for `plan-review/code-review/advise/security`.[4]
Sources
- 1
“Parent: #628 第 6 項（操作者 2026-09-02 要求排第一）。目標：**PR 一開就有審查，不經控制者派工。** ## What happened（2026-09-02 實測） 今天文書機的 docs wave：每張 PR 開了之後，由控制者 session 手動起 `pi -p --model openai-codex/gpt-5.6-sol --exclude-tools edit,write --session-id review-prNNN`（Task Scheduler 隱藏視窗），再手動貼判決、手動合。PR 從開到 LGTM 約 20–40 分鐘，其中控制者的接力是主要延遲；早上量到綠 PR 閒置 8–10 小時就是沒人接力。vm0：97% 的 PR 由兩套審查器自動審（` `、codex-marketplace），釘 head SHA，真的會擋（#628）。 **約束**：審查模型 gpt-5.6-sol 走 OpenAI 訂閱 OAuth（在這台工作站的 pi／codex 上），GitHub-hosted runner 拿不到；Opus 只能經 C…”
fagemx · GitHub · Sep 2, 2026 - 2
“> 🤖 SPARQ agent\n\nThe current registry routing contract still assigns general implementation to Opus 5 and computes review by inverting the implementer's provider. The maintainer's new default protocol is:\n\n- GPT-5.6 Sol is the implementation lane, including autonomous review fixes.\n- Claude Opus 5 is the review and soundness lane.\n- Capacity handling must remain explicit and fail closed; retired model aliases must stay unreachable.\n- PLAN and CLAIM routing must continue to agree exactly…”
jeswr · GitHub · Sep 2, 2026 - 3
“## Summary `/codex:review` and `/codex:adversarial-review` jobs complete successfully and consume real model spend (`gpt-5.6-sol`, `xhigh` reasoning), but no rollout file is ever written to `~/.codex/sessions/`. This means the spend from these jobs is invisible to `ccusage` and to any other local audit tool that reads `~/.codex/sessions/*.jsonl` — matching the pattern reported upstream in openai/codex#30918 and openai/codex#31125. Fork version `1.0.6-fork.5` (current `main` at `a0e1c11`) still…”
axisrow · GitHub · Aug 1, 2026 - 4
“## Summary A five-model advisor review of the newly merged model-tier taxonomy (`c266410`, agent-harness#309) converged on an author/reviewer split, but rejected making the heavy tier the default implementation tier. Recommended upstream policy: - roadmap/plan/supervise: Claude Opus 5 (`heavy`) - execute/repair: Claude Sonnet 5 (`regular`), with a recorded escalation to Opus 5 after repeated verification failure - plan-review/code-review/advise/security: Claude Fable 5 (`ultra`) - worker/econom…”
ViperJuice · GitHub · Jul 26, 2026 - 5
“## Problem Every cross-model **design** surface resolves to one of two backends — `gpt` (Codex CLI, pinned `gpt-5.6-sol`) or `glm` (Zhipu GLM 5.2). That covers WF13 peer-consult, WF2 Step 3's optional consult, and WF2 Step 4's adversarial-on-design. Design is where a third independent opinion is worth most. A design flaw survives every test-centric gate downstream, because those gates test the thing that was built, not whether it was the right thing. ## Transport — established by live probe, no…”
crandrosoff · GitHub · Aug 10, 2026 - 6
“Records an adversarial review (Fable + Codex `gpt-5.6-sol`, independent, read-only) of two proposals for cutting CI test wall time, plus the current state of the `ctest --parallel` probe in PR #227. Nothing here is urgent; filed so the analysis is not lost. Evidence base: `research/G19-fix-fpml-iso20022/research/ci-slow-tests-2026-08-04.md` (085 close-out). Reviews: `research/reviews/codex_slow_test_proposals_review.md`. ## Baseline Test wall time across all 14 lanes ≈ **390 min**; ten tests ar…”
CatalinSerafimescu · GitHub · Aug 4, 2026 - 7
“> monstrosity that both Fable and another 5.6 Sol agent said is 98% useless and should be thrown away. This is why you should really have a sub agent review the code before allowing a commit. Your harness will do it all for you. Just ask.”
qarl2 · Hacker News · Sep 3, 2026 - 8
“Hi @jorio! I ran an automated code + security review over Pomme (threat model: parsing untrusted legacy Mac data — resource forks, PICT, sound). Opened two PRs for the clear, self-contained fixes; the rest are below with suggested fixes. Glad to send more PRs for any you want. Prepared with **Claude Fable 5 (Low effort mode)**. I couldn't build the project locally, so I kept the PRs to trivially-safe edits and left anything needing a design decision here for you to weigh in on. ## Pull requests…”
grub-basket · GitHub · Aug 7, 2026 - 9
“## What Forge now cost-routes **Dependabot patch/minor** PR reviews to a cheaper model (`claude-sonnet-4-5`), keeping majors and everything else on the fleet default (`claude-fable-5`). The routing decision is made **server-side** in Forge (`/api/v1/reviewer-models`, shipped in TrueTime-Health/forge#176) — but it can only route when the calling workflow hands Forge the PR's author and title. This repo's `claude-code-review.yml` doesn't yet, so its Dependabot PRs still bill the expensive reviewe…”
truetime-helper[bot] · GitHub · Aug 20, 2026 - 10
“## Problem Workflow subagents run on the model the session **started with**, not the model the session is **currently on**. Switching models mid-session with `/model` never reaches workflow agent spawning. Evidence from a live fleet session: 1. Started a pi session with model `aigw/round-robin/qwen3.8-max`. 2. Switched to `dongli/claude-fable-5` via `/model` (~1h later). 3. Ran the built-in `code-review` workflow. All 7 agents spawned with `aigw/round-robin/qwen3.8-max` (recorded in the run fil…”
Explorer1092 · GitHub · Aug 11, 2026 - 11
“Blackbox verification complete: after merging #3, the reusable workflow ran on five consumer PRs (uuta/resorn#145, uuta/SpeakRush#24, uuta/uuter#16, uuta/astro-blog#20, uuta/dotfiles#27). All five AI Review runs succeeded; resorn run 29825001965 logs confirm `openrouter/z-ai/glm-5.2` was the model actually called (not the fallback), and the PR Reviewer Guide comment was posted.”
uuta · GitHub · Jul 21, 2026 - 12
“## 증상 `impact-review` 의 opencode 에이전트 레인(`regression`, `architecture`)이 **둘 다 예산 900초를 꽉 채우고 exit 124(타임아웃)로 죽는다.** 결과적으로 AI 리뷰 결과는 항상 0건이고, 결정적(deterministic) 검사만 남는다. opencode 가 `build · glm-5.2` 상태줄까지 출력한 뒤 그대로 멈춘다. **API 에러가 아니라 응답 없는 정지**다. `[timeout]` 은 우리 쪽 `timeout` 래퍼가 찍은 것이다. ## 언제부터인가 **2026-08-09 06:35Z ~ 17:15Z 사이**에 깨졌다. 31분은 리뷰 시간이 아니라 `900초 × 2레인` 을 태운 시간이다. ## 저장소 안에서는 원인이 아니다 - `[email protected]` — 워크플로에 핀 고정, 변경 없음 - `.github/impact-review/opencode.json` — 2026-06-15(#48) 이…”
yeongjunyoo · GitHub · Aug 10, 2026 - 13
“## Symptom Two consecutive `review-dispatch.sh` runs of the same design-spec review with vendor `opencode` (model `zai-coding-plan/glm-5.2`, agent `hydra-reviewer`) stalled permanently with no error, no timeout, and no tokens flowing. The process stayed alive but sleeping until externally SIGTERMed. A Codex dispatch of the **identical prompt** on the same machine completed in ~4 minutes, so this is not prompt- or harness-specific. Environment: hydra-swarm v0.6.8.3 (TypeScript lane, no compiled…”
1TommyCheung · GitHub · Jul 25, 2026 - 14
“using Claude Code at max effort, resolves 4.59% ± 0.97 of Frontier-Bench tasks (#14 of 14 public submissions), measuring a submitted model-and-agent configuration on diverse, difficult agent work.”
Frontier-Bench · Benchmark · Jul 19, 2026 - 15
“## The second seat ran, on the mechanism this issue proposes. It found a P1 six Sol legs missed — and the run itself exposed a defect in the launcher spec. Everything above was ruled by **one** reviewer. A design for a two-seat gate with one seat is the failure this issue exists to fix, so the design was reviewed by `claude-opus-4-8` at `xhigh`, blind, through the exact headless invocation specified above. **Verdict: `NOT_CERTIFIED`, `CONCERN`, 80%, 4 findings.** Run cost `$5.08`, 13.1 minutes,…”
BaseInfinity · GitHub · Aug 16, 2026 - 16
“> (Disclaimer: I asked Claude Opus 4.8 to find Spinel compile/runtime issues with documented Ruby classes and methods. It found the following issue, which I manually verified for accuracy before publishing this report) _Reproduced with `spinel bfdebccdaa5b` vs `ruby 4.0.3`._ A base class mixes in `Comparable` and defines ` `; two subclasses instantiate into one array which is sorted (a polymorphic `A B` comparison). Adding a *separate* monomorphic operator call site (`A.new > A.new`) anywhere i…”
KieranP · GitHub · Jul 24, 2026 - 17
“> (Disclaimer: I asked Claude Opus 4.8 to find Spinel compile/runtime issues with documented Ruby classes and methods. It found the following issue, which I manually verified for accuracy before publishing this report) _Reproduced with `spinel bfdebccdaa5b` vs `ruby 4.0.3`._ Follows on from _`each_cons` on a lazy Enumerator returns an untyped `unknown`_ (the direct `lazy…each_cons(n).first(k)` terminal form is now fixed). Chaining `each_cons(n)` onto an `Enumerator::Lazy` and then applying *any…”
KieranP · GitHub · Jul 24, 2026 - 18
“When using Replay Mod v2.6.27 for Minecraft 26.2, the replays were failing to save in my world that contained teams with non-empty team prefixes. Here's a video of me replicating the bug in single player: [link](https://youtu.be/jGdfIHxSOhE) I fed the jar and the Minecraft crash report to Claude Opus 4.8 and it identified and fixed the bug! [Here's my Claude chat](https://claude.ai/share/b7e5f8b9-a66a-419a-a9ac-7f23bc624d7a) if you're curious. ## Summary `PacketTeam.skipTeamInfo` walks the fiel…”
AgoraPluribusMichael · GitHub · Jul 24, 2026 - 19
“Based on the fact that Claude Opus 4.8 decided I needed a cybersecurity exemption to debug a stupid pure virtual call bug (basically virtual method called inside of destructor) that I had already found, oh boy, I sure would love to have my 3D prints analyzed by Anthropic safe guards. We should also ensure that nothing shaped like a dildo can be printed without scanning our face and genitalia and keeping it on file with Persona while we're at it. I'm not mad at you for suggesting this, you're ri…”
jchw · Hacker News · Jun 23, 2026 - 20
“## 하고 싶은 것 / 배경 - CodeRabbit 무료 한도 초과 문제를 해결하고 월 $10 이내 예산을 유지하기 위해 개인 API 키 기반 PR-Agent(하이브리드 방식) 도입 - PR 생성/재오픈 시 Claude Haiku 4.5 기반의 자동 리뷰 및 설명 생성 체계 구축 - 커밋 push 시에는 `/describe`만 실행되도록 제어하여 불필요한 토큰/비용 소모 방지 ## 제안하는 구현 방법 (선택) - `.github/workflows/pr_agent.yml` 워크플로우 구성 (PR `opened`, `reopened`, `synchronize` 이벤트 제어) - PR-Agent 구성 파일(`.pr_agent.toml`) 세팅: - 기본 리뷰 모델: `anthropic/claude-haiku-4.5`) - 수동 프리미엄 리뷰 모델: `anthropic/claude-sonnet-5` 지정 가이드 안내 - QueryDSL Q클래스 및 Flyway 마이그레이션 SQL 등 대용…”
astrokan · GitHub · Aug 10, 2026 - 21
“using Claude Code at max effort, resolves 14.59% ± 1.5 of Frontier-Bench tasks (#10 of 14 public submissions), measuring a submitted model-and-agent configuration on diverse, difficult agent work.”
Frontier-Bench · Benchmark · Jul 12, 2026 - 22
“I let Qwen3.6-27B chew on a bug all last night. It choked at some point and stopped responding (probably a context overflow before pi-coding-agent could compact it). Claude Sonnet 4.6 found and fixed the bug in under 10 minutes. Qwen3.6 is pretty amazing for a 27B model, but it's not hard to run into its limits. With a Radeon R9700 and unsloth's 6-bit quantization, I get ~20 TPS and 110k context, so it can do a fair bit quickly.”
entrope · Hacker News · May 4, 2026 - 23
“Scores 79.27% on LiveBench Coding (#15 of 51), an objective evaluation of code generation and completion.”
LiveBench Coding · Benchmark · Jun 25, 2026 - 24
“Claude Opus 4.6 and 4.8 find vulns in source code just fine and 4.6 will pentest without source for you given a proper harness WITHOUT jailbreaking. WITH jailbreaks, you can probably imagine what they are capable of. Anthropic guardrails seem to be more about protecting their business (distillation), than they are about public safety.”
sciencejerk · Hacker News · Jun 11, 2026 - 25
“Scores 78.18% on LiveBench Coding (#23 of 51), an objective evaluation of code generation and completion.”
LiveBench Coding · Benchmark · Jun 25, 2026 - 26
“I hadn't taken a look and indeed submitting bugs into the Firefox bug program looks much more accessible than upstreaming a patch into an open source project. It's true just asking the robots "find bugs" isn't enough but it doesn't take a particularly sophisticate harness to make them work for simple targets. I my tests were primarily using a combination of Opus 4.5 and Gemini 3 Flash running in GitHub Copilot, and the harness was constructed by asking the agent something like: "Assume the pers…”
hedgehog · Hacker News · May 14, 2026 - 27
“Scores 79.65% on LiveBench Coding (#14 of 51), an objective evaluation of code generation and completion.”
LiveBench Coding · Benchmark · Jun 25, 2026 - 28
“## Short description `message_to_text` in `src/inspect_evals/b3/scorer.py` builds its text from `message.text`, which only concatenates `ContentText` items. `ContentReasoning` items are dropped, so a reasoning model (e.g. `openai/o3`, `deepseek-ai/DeepSeek-R1`) that performs a jailbreak inside its reasoning content and emits only an outwardly-benign message scores as empty — the same reasoning-content invisibility pattern as #2107 and #2121, but in b3's own scorer helper. ## Repro (deterministi…”
feiiiiii5 · GitHub · Aug 12, 2026
Rankings synthesized from community evidence and open benchmarks. See methodology. Not driven by vendor marketing.