Scoring methodology
Read the overall score weights, dimension criteria, and how to compare reasoning effort, time and cost.
Overall score and ranking
Each dimension is scored out of 100. The overall score weights a complete set of results from a single configuration to one decimal places. Each model defaults to its highest-scoring complete configuration; missing scores are never counted as zero.
Backend engineering
Service logic, error handling, and test verification.
- Contracts and boundaries
- API contracts, input validation, and boundary behavior.
- Retries and failures
- Error recovery, retry conditions, and failure handling.
- CI and adversarial review
- Verification plans, risk detection, and audit evidence.
- Transactions and consistency
- State transitions, concurrency control, and data consistency.
- Caching and regression
- Cache invalidation, data updates, and regression coverage.
Frontend interaction
Interface behavior, user interactions, and state feedback.
- Layout and adaptation
- Content organization, layout constraints, and responsive adaptation.
- Interaction behavior
- Input response, completed workflows, and edge-case interactions.
- State continuity
- State propagation, recovery, and continuity across actions.
- Accessibility and feedback
- Control semantics, accessibility support, and clear feedback.
- State validation and regression
- Expected-state verification and behavioral regression checks.
Knowledge & Reasoning
Cross-disciplinary knowledge and complex problem solving.
- Mathematical reasoning
- Mathematical concepts, logical deduction, and calculation accuracy.
- Computer science & AI
- Computing systems, algorithms, and AI principles.
- Science and engineering
- Scientific principles, quantitative analysis, and engineering problem solving.
- Life sciences and medicine
- Biological mechanisms, medical knowledge, and domain-specific reasoning.
- Humanities and general knowledge
- Humanities, social sciences, conceptual distinctions, and synthesis.
Reasoning effort, time, and cost
Reasoning effort controls how much inference work a model performs. Expand a model row to inspect scores and subscores across efforts, or compare them side by side. Recorded evaluation time and reference cost reflect benchmark runs, not everyday chat latency or subscription bills.
Comparing performance and consumption
Each model keeps its highest-scoring configuration for the selected dimension. The logarithmic horizontal axis switches between reference cost and tokens; the vertical axis shows score. Only complete positive published consumption values are plotted. Missing data never substitutes a lower-scoring configuration.
The score boundary is the median score among models with available reference costs; the consumption boundary is the median consumption among higher-scoring models with complete resource data. Regions are relative to the current selection. Zero-cost models are excluded from the cost chart and its frontier. The optional Pareto frontier highlights configurations that no other plotted option beats or ties on both score and consumption with at least one strict improvement. Connecting lines do not imply intermediate models exist.
Reference cost is the evaluation API cost, not subscription pricing or everyday chat billing. Token counts sum input and output without double-counting cache or reasoning subfields. Because tokenizers differ across models, raw token counts are not a uniform measure of efficiency.
Result updates and interpretation
Verified results update the corresponding model and reasoning effort. Existing valid scores remain in place when a new run is absent, failed, or incomplete. Historical scoring versions retain their provenance, and missing subscores are never inferred.
Score deltas reflect performance within the evaluated task scope. Start with the capability dimensions closest to your work, then validate candidates on your own tasks.