Our methodology
Trust the process.
Then inspect it.
This field guide is automated. Hardware affiliate links are disclosed and sit outside the ranking, which makes the method, limits, and audit trail more important, not less.
end-to-end process
Six gates from raw signal to a published answer
- 01
Collect
Catalog facts, benchmarks, prices, and public practitioner reports.
- 02
Classify
Associate evidence with the exact model, task, sentiment, and source type.
- 03
Score
Apply the use case's fixed weights to current, eligible model data.
- 04
Draft claims
Propose atomic tips and cautions from the supplied evidence.
- 05
Verify
Audit each claim once and discard unsupported output.
- 06
Publish
Replace the complete recommendation atomically and rebuild the site.
Ranking, claim drafting, and verification are separate stages. The drafting model never receives authority to reorder candidates. A separate model audits atomic claims once; unsupported claims are discarded, and deterministic code assembles the published prose.
source observatory
What the pipeline can actually see
45 of 55 configured or intended sources are healthy. Disabled sources stay in task coverage as visible gaps.
| Source | Kind | Status | Rows matched | Last success |
|---|---|---|---|---|
| Aider Polyglot | benchmark | healthy | 37/69 | 2026-10-20 |
| Berkeley Function Calling | benchmark | healthy | 47/109 | 2026-10-20 |
| design arena | benchmark | healthy | 435/498 | 2026-10-20 |
| FrontierMath Tiers 1-3 (Epoch AI) | benchmark | healthy | 102/120 | 2026-10-20 |
| GPQA Diamond (Epoch AI) | benchmark | healthy | 193/324 | 2026-10-20 |
| OTIS Mock AIME (Epoch AI) | benchmark | healthy | 193/304 | 2026-10-20 |
| SimpleQA Verified (Epoch AI) | benchmark | healthy | 75/91 | 2026-10-20 |
| FACTS Grounding | benchmark | healthy | 45/52 | 2026-10-20 |
| Terminal-Bench (former Frontier-Bench) | benchmark | disabled | 0/0 | 2026-10-01 |
| JevBench | benchmark | healthy | 116/116 | 2026-10-20 |
| LiveBench | benchmark | failing | 0/0 | 2026-10-18 |
| LMArena Agent | benchmark | healthy | 50/53 | 2026-10-20 |
| LMArena Agent Bash Recovery | benchmark | healthy | 50/52 | 2026-10-20 |
| LMArena Agent Steerability | benchmark | healthy | 50/53 | 2026-10-20 |
| LMArena Agent Task Outcome | benchmark | healthy | 50/53 | 2026-10-20 |
| LMArena Agent Tool Hallucination | benchmark | healthy | 50/53 | 2026-10-20 |
| LMArena WebDev | benchmark | healthy | 112/147 | 2026-10-20 |
| LMArena Creative Writing | benchmark | healthy | 174/414 | 2026-10-20 |
| LMArena Document | benchmark | healthy | 40/44 | 2026-10-20 |
| LMArena Instruction Following | benchmark | healthy | 174/416 | 2026-10-20 |
| LMArena Long Query | benchmark | healthy | 174/394 | 2026-10-20 |
| LMArena Math | benchmark | healthy | 169/400 | 2026-10-20 |
| LMArena Search | benchmark | healthy | 13/42 | 2026-10-20 |
| LMArena Search Factuality | benchmark | healthy | 13/40 | 2026-10-20 |
| LMArena Text | benchmark | healthy | 173/416 | 2026-10-20 |
| LMArena Text Factuality | benchmark | healthy | 128/185 | 2026-10-20 |
| LMArena Vision | benchmark | healthy | 96/167 | 2026-10-20 |
| LMArena Vision Diagrams | benchmark | healthy | 89/123 | 2026-10-20 |
| LMArena Vision OCR | benchmark | healthy | 89/123 | 2026-10-20 |
| LMArena WebDev Image-to-Web | benchmark | healthy | 53/64 | 2026-10-20 |
| OSWorld Verified | benchmark | healthy | 5/9 | 2026-10-20 |
| SWE-bench Verified | benchmark | healthy | 11/18 | 2026-10-20 |
| SWE-rebench | benchmark | healthy | 12/13 | 2026-10-20 |
| tau2-bench | benchmark | healthy | 22/27 | 2026-10-20 |
| Terminal-Bench | benchmark | healthy | 35/35 | 2026-10-20 |
| Bluesky | community | disabled | 0/0 | never |
| GitHub Discussions | community | degraded | 205/205 | 2026-10-20 |
| Hacker News | community | healthy | 3893/3893 | 2026-10-20 |
| Hugging Face discussions | community | healthy | 12/12 | 2026-10-20 |
| Lemmy | community | healthy | 62/62 | 2026-10-20 |
| CRAG | owned eval | disabled | 0/0 | never |
| LongBench v2 | owned eval | disabled | 0/0 | never |
| Route reliability | owned eval | disabled | 0/0 | never |
| Structured-output eval | owned eval | disabled | 0/0 | never |
| VLMEvalKit tasks | owned eval | disabled | 0/0 | never |
| WMT translation | owned eval | disabled | 0/0 | never |
| GitHub issues | practitioner | healthy | 373/373 | 2026-10-20 |
| RSS · https://aider.chat/feed.xml | practitioner | healthy | 0/10 | 2026-10-20 |
| RSS · https://huggingface.co/blog/feed.xml | practitioner | healthy | 0/882 | 2026-10-20 |
| RSS · https://metr.org/feed.xml | practitioner | healthy | 1/98 | 2026-10-20 |
| RSS · https://simonwillison.net/atom/everything/ | practitioner | healthy | 5/30 | 2026-10-20 |
| RSS · https://thezvi.substack.com/feed | practitioner | healthy | 0/20 | 2026-10-20 |
| RSS · https://www.interconnects.ai/feed | practitioner | healthy | 0/20 | 2026-10-20 |
| RSS · https://www.latent.space/feed | practitioner | healthy | 3/20 | 2026-10-20 |
| Stack Exchange | practitioner | healthy | 4/4 | 2026-10-20 |
- 01
Catalog, not copy
OpenRouter and models.dev supply language-model facts; the ranked JevBench roster supplies typed decision systems. Chat baselines, raw-logit controls, classifiers and rerankers in that roster are excluded from the decision catalog. Separate checkpoint variants keep separate identities. These sources supply structured facts such as pricing, context, modalities, release dates, capabilities, and third-party benchmark identifiers. Newly cataloged models are joined automatically to benchmark rows by permanent slug. We never republish vendor descriptions.
- 02
Evidence has a type
Benchmarks, independent evaluators, practitioner reports, community discussions, operational telemetry, and official release evaluations are kept distinct. JevBench, LiveBench, BFCL, SWE-rebench, Terminal-Bench, OSWorld, tau2-bench, FACTS Grounding, Epoch AI, LMArena, Design Arena, Aider, SWE-bench, and configured task suites retain their evaluator and harness lineage. First-party release claims can never count as independent confirmation.
- 03
The question sets the weights
There is no universal best model. Each use case defines its own task-specific benchmark, price, context, reliability, and ownership signals. Scores average only measured inputs. Popularity, release age, community volume and missing-test penalties add no points. Only the newest coherent benchmark snapshots participate.
- 04
Coverage has a named status
A model with one excellent result should not automatically equal one tested broadly. Coverage is measured against every intended feed, including a configured feed that is failing or not yet available. Established measurements require at least two independent evaluator families. Coverage describes breadth, not a probability that the model is best. Close scores are labelled without claiming statistical significance, and among close leaders the one measured by more evaluator families ranks first. One independent result or official release evidence is provisional.
- 05
The writer does not rank
The deterministic scorer owns candidate order. A language model turns supplied evidence into readable verdicts but cannot promote a model over the computed order.
- 06
Every source is checked
Citations are checked for model ownership and source existence, then a separate validation pass tests whether each excerpt supports the prose. Claims are attributed to the exact model and task, with integration failures separated from capability. One conversation counts once and each author has a capped contribution. First-hand reports decay with a 45-day half-life; neutral mentions add no endorsement. Evaluators, repositories, publications, communities, and negative reports retain separate lineage. Unsupported output fails closed.
- 07
Freshness is explicit
Catalog, every benchmark source, official vendor release discovery, evidence, and synthesis each refresh daily. New model announcements are found through reviewed first-party sitemaps and exact model identity checks. Relevant evidence, material catalog changes, new benchmark snapshots, or the freshness ceiling trigger regeneration for affected use cases. Discovery rotates through measured challengers with few practitioner reports regardless of release age. Observation dates and evaluation versions are kept separate.
- 08
Constraints rerank a wider field
Editorial pages stay concise, but synthesis retains a larger pool of citation-audited candidates. Picker constraints such as open weights, price, context, modalities, and deployment route deterministically rerank that pool. A one-sided candidate can appear there only when its supported evidence is shown and the missing side is stated plainly.
- 09
One answer survives failures
Every selectable use case has one explicit top recommendation. If an upstream source or synthesis run fails, the last completely verified answer remains published and the pipeline raises an operational alert. A failed source retains its last verified snapshot with its last successful check shown. Benchmark identity or cohort mismatches are excluded; contaminated historical versions are never revived.
who runs this
An automated publication
No person edits the rankings. The pipeline above collects, scores, drafts and verifies on a schedule, and every published claim links to its source. The operator maintains the code and the source list, and earns only from the disclosed hardware affiliate link, which never enters the ranking.
corrections policy
How errors get fixed
- A source deleted at its origin is removed from the next build, and claims that relied on it are dropped.
- A wrong number or attribution is fixed in the generator, not by hand on one page, so every page using that rule is corrected together.
- Each ranking shows its revision number and update date, so a correction is visible as a new revision.
known limits
A useful map is still not the territory.
Community evidence leans developer-heavy, public discussion can be promotional or mistaken, benchmark coverage varies, and listed API prices do not capture hosting or latency. Official release evidence closes the launch-day information gap but remains visibly provisional until independent sources reproduce relevant results. Treat the top pick as the strongest model to test first, then validate it against your prompts, latency target, and failure budget.