In-Depth Comparison and Benchmarks of eustella v1.5
Comparing eustella v1.5 on our selected benchmarks across proprietary frontier models and open-weight models.
This is the second of three research articles we are releasing in connection with eustella Version 1.5. The evaluations reported here were already part of our internal model-evaluation process; we publish them now because we believe that a sovereign AI stack is only as trustworthy as its evaluations are open. We are publishing the method and the numbers so that the results can be examined, evaluated, and criticised. More, and deeper, evaluations will follow.
You can find all the benchmarks, sources and references cited at the end of this article.
Summary
eustella v1.5 is the new AI system we developed for our agent platform, eustella. The new version includes an optimised harness, weight quantisations with near-zero quality loss, and RL and other post-training steps. Our optimisations can be applied to almost all open-weight models. For this comparison we show eustella v1.5 on the base model Qwen3.8-27B. We compared eustella v1.5 against 37 published models from many different labs on the market, in six comparison groups (its base model, open-weight models up to 35B, open-weight models above 35B, Mistral, OpenAI and Anthropic), on the fifteen benchmarks of our evaluation set. The two models in the whole set that we ran ourselves — eustella v1.5 and the unmodified Qwen3.8-27B base weights, under the same inference settings — are the only like-for-like numbers in this article (on two judge-graded benchmarks the judges differ; see Limitations); every other model is a published number we collected and attributed. The comparisons name the source of each external number: model developer (lab), benchmark organisation, or community run. Each external number was checked against its source, not reproduced by us.
Because most published model cards do not cover all fifteen benchmarks, the number of shared benchmarks (n) differs from model to model. A win/loss comparison is only as meaningful as the number of benchmarks shared in the comparison.
Overview: eustella v1.5 against all 37 models
Each row is one model: eustella v1.5's wins and losses on the benchmarks that model reports, with n the number of shared benchmarks. A lead over many shared benchmarks is stronger evidence than a lead over two or three, so rows with fewer than five are shown faded. The scores behind every row are in that group's section below.
- eustella v1.5 loses
- eustella v1.5 wins
- nshared benchmarks; under 5 is thin evidence
A short list of widely known models.
All sources for these numberseustella v1.5 against its base model, Qwen3.8-27B
Against the unmodified base model, Qwen3.8-27B, run by us under identical settings, eustella v1.5 leads on all 14 shared benchmarks. That is the only comparison in this article that can be compared like for like. The differences are largest on instruction following (IFBench 82.6 vs 71.67), code (LiveCodeBench v6 89.1 vs 72.0; the base-model figure is a floor, as 22.9% of its answers hit the 48k-token cap), and the honesty metric (AA-Omniscience −3.4 vs −11.17, graded by different judges on the two runs). Alibaba's official model card reports only three shared benchmarks: eustella v1.5 leads on GPQA Diamond and IFBench and trails on LiveCodeBench v6 (89.1 vs 90.3). Against the widest published set from any source we found eight shared benchmarks. In those eight, eustella v1.5 loses only two: LiveCodeBench v6 to Alibaba's own card, and AIME 2026 to a third-party run of the same weights (Aleph Alpha, 97.7).
GPQA Diamond acc, avg@4
IFBench prompt-level strict
LiveCodeBench v6 pass@1
AIME 2026 acc, avg@8
MATH-500 pass@1
MMLU-Redux 2.0 acc
IFEval prompt-level strict
HumanEval+ pass@1
Global-MMLU-Lite de acc
Global-MMLU-Lite en acc
AA-Omniscience Omniscience index
AA-LCR v1.1 acc, avg@3
MultiChallenge overall
MMLU-ProX-Lite de acc
WMT24++ en→de COMET-22
| Model | W–L–T (n) | GPQA Diamond | IFBench | LiveCodeBench v6 | AIME 2026 | MATH-500 | MMLU-Redux 2.0 | IFEval | HumanEval+ | Global-MMLU-Lite de | Global-MMLU-Lite en | AA-Omniscience | AA-LCR v1.1 | MultiChallenge | MMLU-ProX-Lite de | WMT24++ en→de |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| eustella v1.5 | 92.57 | 82.6 | 89.1 | 93.5 | 97.7 | 92.6 | 93.1 | 95.1 | 91.5 | 92.5 | -3.4 | 87.5 | 74.3 | 86.1 | 0.832 | |
| Qwen3.8-27B, bare (our run) | 14–0–0 (14) | 85.5own | 71.67own | 72own | 92.75own | 95.7own | 91.24own | 91.4own | 92.43own | 88.5own | 90.75own | -11.17own | 63.55own | 83.67own | 0.829own | |
| Qwen3.8-27B, published | 6–2–0 (8) | 89.2L | 79.5L | 90.3L | 97.7L | 91.19L | 94.7L | -10B | 82B |
Bold: higher than eustella v1.5. Sources: own = our run, L = lab, B = benchmark organisation, C = community run. W–L–T is eustella v1.5's wins, losses and ties; n is the number of shared benchmarks.
eustella v1.5 against the open-weight sub-35B class
eustella v1.5 leads in its own class on every head-to-head: nine models, with the shared-set tallies running from 2–0 to 9–1. The losses are spread thinly: MMLU-Redux 2.0, where Qwen3.6 at both sizes and Gemma 4 31B sit at 93.3–93.7 against our 92.6; two AIME 2026 results (Qwen3.6-27B and Muse Glimmer 30B); three IFEval results (Gemma 4 31B, Gemma 4 26B-A4B and Nemotron-3.5-Lightning-30B, each with the grader variant unstated; see Limitations), and Global-MMLU-Lite German, where Gemma 4 31B edges ahead (91.75 vs 91.5). On the honesty metric the spread across the class is wide (−17.7 to −51.6) against our −3.4.
GPQA Diamond acc, avg@4
+7 more
IFBench prompt-level strict
+6 more
LiveCodeBench v6 pass@1
+4 more
AIME 2026 acc, avg@8
+5 more
MATH-500 pass@1
+1 more
MMLU-Redux 2.0 acc
+1 more
IFEval prompt-level strict
+1 more
HumanEval+ pass@1
+1 more
Global-MMLU-Lite de acc
Global-MMLU-Lite en acc
AA-Omniscience Omniscience index
+6 more
AA-LCR v1.1 acc, avg@3
+6 more
MultiChallenge overall
eustella v1.5 against the large open-weight set
Eighteen open-weight models above 35B parameters, fifteen of which share at least four benchmarks with eustella v1.5. eustella v1.5 wins the head-to-head against thirteen of those fifteen. The two that hold the top of the open field are where it loses: Kimi K3 takes four of seven shared benchmarks (GPQA Diamond 93.5, AIME 2026 96.67, AA-Omniscience 19.7, AA-LCR 88.67), and Qwen3.8-Max, the API version of the open Qwen3.8-2.4T-A95B, takes three of four, two of them by a hair (GPQA Diamond 92.6 against our 92.57, IFBench 82.8 against 82.6) and the third on AA-Omniscience (11.98). Solar Open2 250B takes three of seven (LiveCodeBench v6, AIME 2026, AA-Omniscience) and Inkling two of six (AIME 2026, AA-Omniscience). GLM-5.3, GLM-5.3-Flash and Nemotron 3 Ultra each take exactly one benchmark, the honesty metric; Qwen3.5-397B-A17B takes only MMLU-Redux 2.0, and Command A+, Ring-2.6-1T and Inkling-Small only AIME 2026.
GPQA Diamond acc, avg@4
+14 more
IFBench prompt-level strict
+13 more
LiveCodeBench v6 pass@1
+11 more
AIME 2026 acc, avg@8
+6 more
MATH-500 pass@1
+1 more
MMLU-Redux 2.0 acc
IFEval prompt-level strict
+2 more
Global-MMLU-Lite de acc
AA-Omniscience Omniscience index
+15 more
AA-LCR v1.1 acc, avg@3
+16 more
MultiChallenge overall
+5 more
eustella v1.5 against Mistral
Sadly we were not able to fully compare eustella v1.5 against Mistral's newest model, Mistral Large 4, since eustella v1.5 shares only three benchmarks with the ones published for Large 4 so far. Its closest margin of those three, AA-Omniscience (−3.4 vs −5.3), is not a like-for-like comparison: our −3.4 was graded by GPT-5.6 Luna in our own setup, while Mistral's −5.3 is Artificial Analysis's official grading.
GPQA Diamond acc, avg@4
IFBench prompt-level strict
LiveCodeBench v6 pass@1
AIME 2026 acc, avg@8
AA-Omniscience Omniscience index
AA-LCR v1.1 acc, avg@3
MultiChallenge overall
| Model | W–L–T (n) | GPQA Diamond | IFBench | LiveCodeBench v6 | AIME 2026 | AA-Omniscience | AA-LCR v1.1 | MultiChallenge |
|---|---|---|---|---|---|---|---|---|
| eustella v1.5 | 92.57 | 82.6 | 89.1 | 93.5 | -3.4 | 87.5 | 74.3 | |
| Mistral Large 4 (Preview) few shared | 3–0–0 (3) | 88.1B | -5.3B | 81.33B | ||||
| Mistral Medium 3.5 | 7–0–0 (7) | 74.8B | 68.8B | 84.9L | 89L | -37B | 69B | 49.8L |
Bold: higher than eustella v1.5. Sources: own = our run, L = lab, B = benchmark organisation, C = community run. W–L–T is eustella v1.5's wins, losses and ties; n is the number of shared benchmarks.
eustella v1.5 against OpenAI
eustella v1.5 loses the head-to-head to both closed frontier models — GPT-6.1 Sol on four of seven (GPQA Diamond, MATH-500, AA-Omniscience, MultiChallenge) and GPT-6 Sol on three of four — and leads OpenAI's own open-weight gpt-oss-120b on all seven shared benchmarks. Of the previous generation, eustella v1.5 leads GPT-5.6 Luna on seven of eight shared benchmarks (losing only AIME 2026) and loses narrowly to GPT-5.6 Sol, four wins to five losses.
GPQA Diamond acc, avg@4
+2 more
IFBench prompt-level strict
+1 more
LiveCodeBench v6 pass@1
AIME 2026 acc, avg@8
MATH-500 pass@1
+3 more
Global-MMLU-Lite de acc
+1 more
Global-MMLU-Lite en acc
+1 more
AA-Omniscience Omniscience index
+2 more
AA-LCR v1.1 acc, avg@3
+3 more
MultiChallenge overall
eustella v1.5 against Anthropic
eustella v1.5 leads Claude Haiku 5.5 on five of seven shared benchmarks and edges Claude Opus 5.5 over the small shared set it has (two wins, one loss on AA-Omniscience, one tie on HumanEval+). All three Anthropic models lead it on the honesty metric. Against the previous Claude Opus 5 it is level at four wins and four losses; three of those wins (LiveCodeBench v6, MATH-500, HumanEval+) are by 0.1–0.6 points, while Opus 5 leads clearly on German and English Global-MMLU-Lite and on the honesty metric.
GPQA Diamond acc, avg@4
+1 more
LiveCodeBench v6 pass@1
MATH-500 pass@1
HumanEval+ pass@1
Global-MMLU-Lite de acc
Global-MMLU-Lite en acc
AA-Omniscience Omniscience index
AA-LCR v1.1 acc, avg@3
+1 more
MultiChallenge overall
eustella's current weaknesses
The two metrics on which eustella v1.5 loses most often across the whole field are AA-Omniscience (the honesty/overclaiming index), where 16 of 36 models score higher, and competition math, where 11 of the 20 models reporting AIME 2026 score higher. eustella v1.5 overclaims more often than the frontier models do; our post-training is aimed at reducing that next.
Limitations — what this is, and what it is not
What this is not: a claim that eustella v1.5 is a SOTA frontier model, or that any model can be made to lead on every benchmark. It is a published comparison of one configuration against 37 models currently on the market, with our evaluation framework of fifteen benchmarks.
Single instrument. All of our numbers come from one harness (evalscope 1.10.0) at one set of sampling settings, at one reasoning effort. Where we could not attach a confidence interval to a run, the number is a single sample. Where a benchmark's own grader or its full set was not used, we say so here: MultiChallenge was self-judged for eustella v1.5 and judged by GPT-4o for the base model; AA-Omniscience was graded by GPT-5.6 Luna on a 149-item subset for eustella v1.5 and by a different judge for the base model, while every external AA-Omniscience number is Artificial Analysis's official grading; eustella v1.5's AA-LCR score comes from a 32-item subset and its IFBench score from a 109-item subset. The large-effect deltas (the base-model lead, the sub-35B lead, the gpt-oss-120b lead) are too large to be an evaluation artefact; the smaller effects (the IFEval losses to Gemma and Nemotron, the MMLU-Redux losses to Qwen 3.6 and Gemma) are judge- and harness-sensitive and should be read at that level.
The external numbers are not ours. Every number that is not one of our two runs is a published figure from a lab card, a benchmark organisation, or a community run, taken as reported, checked against its source and not reproduced by us. A lab's self-reported GPQA can differ from independent runs of the same weights by several points (Qwen3.8-27B: 89.2 on the card vs 90.5 at Artificial Analysis; Qwen3.6-27B: 87.8 on the card vs 84.24 at Artificial Analysis and 85.9 at Epoch AI), and we do not reconcile them. Where sources conflict strongly we keep both and name it: Vals' MATH-500 for gpt-oss-120b (94.8) against Kaggle's (76.53); Liquid's MATH-500 for Qwen3.6-35B-A3B (96.73) against RedHat's (84.8). The win/loss tallies use one number per cell, chosen at the point of use and noted in the sources.
The matching benchmark number varies. A 9-to-1 tally (Qwen3.6-35B-A3B) is not the same evidence as a 3-to-0 tally (Mistral Large 4). Where the shared set is under five benchmarks we have labelled the result. The Claude Opus 5.5 (n = 4) and Mistral Large 4 (n = 3) comparisons are the weakest named in this article.
Metric-variant ambiguity. Several external IFEval, IFBench and IFEval-strict numbers do not state whether the grader is prompt-level or instruction-level, or strict or loose. We use the number as the source states it and flag the missing variant; a strict-vs-loose difference can move a number by several points, so the IFEval losses to Gemma 4 31B (94.58), Gemma 4 26B-A4B (98.5), Nemotron-3.5-Lightning-30B (94.19) and Qwen3.5-122B-A10B (93.4) should be read with that ambiguity attached.
Not production traffic. These are benchmark runs, not production behaviour. A benchmark lead does not transfer to deployment on a user's real distribution of queries; the political-bias article is the one that measures deployed-behaviour deltas.
One language pair. The German-language results rest on one language pair (WMT24++ en→de) and one per-language set each (Global-MMLU-Lite de, MMLU-ProX-Lite de). No external model publishes MMLU-ProX-Lite de or WMT24++ en→de on its own, so those two have no external comparison. It is not a general multilingual claim.
Methodology
Inference setup
Two configurations were run by us, on the same hardware, with the same vLLM build and the same flags, so that the base-model delta is a like-for-like measurement. All other models are published numbers and were not run by us.
eustella v1.5
- Weights: eustella v1.5, also quantised for a 1:1 comparison with the base model
- vLLM: 0.25.1
- Flags:
--kv-cache-dtype fp8 --max-model-len 262144 --max-num-seqs 32 --max-num-batched-tokens 32768 --enable-prefix-caching --enable-chunked-prefill --async-scheduling --speculative-config '{"method":"mtp","num_speculative_tokens":2}' --generation-config auto --reasoning-parser qwen3
Unmodified Qwen3.8-27B (base model)
- Weights: bare Qwen3.8-27B, quantised
- vLLM: 0.25.1
- Flags: identical to eustella v1.5
Sampling (identical for base model and eustella v1.5)
- temperature 1.0, top_p 0.95, top_k 20
- reasoning effort: xhigh
- harness: evalscope 1.10.0
Benchmarks
Fifteen benchmarks, in each benchmark's own terms: GPQA Diamond (accuracy, avg@4); MATH-500 (pass@1); AIME 2026 (accuracy, avg@8); MMLU-Redux 2.0 (accuracy); Global-MMLU-Lite German and English (accuracy); MMLU-ProX-Lite German (accuracy); WMT24++ en→de (COMET-22, 0–1); HumanEval+ and LiveCodeBench v6 (pass@1); IFEval and IFBench (prompt-level strict); MultiChallenge (overall, 0–100); AA-LCR v1.1 (accuracy, avg@3); AA-Omniscience (index, −100 to 100).
Conclusion
eustella v1.5 runs on a small dense 27B model. It leads its own base model on all 14 shared benchmarks, leads every model in its size class head to head, and wins the head-to-head against most open models several times its size, among them Qwen3.5-397B-A17B and gpt-oss-120b. It trails the current closed frontier (GPT-6.1 Sol and GPT-6 Sol, and on the honesty metric every Anthropic model) and the strongest open releases (Kimi K3, Qwen3.8-Max), mostly on the honesty metric and competition math. Its German-language results lead its base model on all three German benchmarks, but have little external comparison: on Global-MMLU-Lite German it trails Claude Opus 5, GPT-5.6 Sol and Gemma 4 31B, and no other model publishes MMLU-ProX-Lite German or WMT24++ en→de.
All references and sources
Every published score, its source type (lab, benchmark organisation or community run) and its URL, with the notes on conflicting sources and unstated metric variants. Source types: lab = the model developer, or another lab's model card that uses the model as a baseline; benchmark org = Epoch AI, Artificial Analysis, Kaggle, Scale or Vals; community = a third-party run.
Base model 2 models
Qwen3.8-27B, bare (our run)
- GPQA Diamond85.5our run
- IFBench71.67our run
- LiveCodeBench v672our run
- AIME 202692.75our run
- MATH-50095.7our run
- MMLU-Redux 2.091.24our run
- IFEval91.4our run
- HumanEval+92.43our run
- Global-MMLU-Lite de88.5our run
- Global-MMLU-Lite en90.75our run
- AA-Omniscience-11.17our run
- MultiChallenge63.55our run
- MMLU-ProX-Lite de83.67our run
- WMT24++ en→de0.829our run
Qwen3.8-27B, published
- GPQA Diamond89.2labhuggingface.co/Qwen/Qwen3.8-27B
- IFBench79.5labhuggingface.co/Qwen/Qwen3.8-27B
- LiveCodeBench v690.3labhuggingface.co/Qwen/Qwen3.8-27B
- AIME 202697.7labhuggingface.co/Aleph-Alpha/Kolibri-1
- IFEval91.19labhuggingface.co/RedHatAI/Qwen3.8-27B-MXFP4
- HumanEval+94.7labhuggingface.co/Aleph-Alpha/Kolibri-1
- AA-Omniscience-10benchmark orgartificialanalysis.ai/models/qwen3-8-27b
- AA-LCR v1.182benchmark orgartificialanalysis.ai/models/qwen3-8-27b
All published numbers for the official Qwen/Qwen3.8-27B at xhigh, from any source; Alibaba's official card alone reports only Alibaba's own three. GPQA/IFBench/LCB v6 from Alibaba's card; AA-LCR/Omniscience from AA (AA GPQA 90.5). AIME 2026 (EN; DE 96.9) and HumanEval+ are Aleph Alpha's baseline column on the Kolibri-1 card (same table: LCB v6 93.8, AA-LCR 81.3, IFBench 81.9 loose, Omniscience -5.8, MMLU-ProX CoT DE 82.4 — full ProX, not Lite, not included). IFEval 91.19 (prompt strict) is RedHat's run on its MXFP4 card; its MATH-500 83.67 ran with a 32k token cap and is left out. Our earlier comparison's unsourced community numbers (MATH-500 95.2, IFEval 89.7, HumanEval+ 92.1) are left out.
Open-weight, up to 35B 9 models
Qwen3.6-35B-A3B
- GPQA Diamond84.1benchmark orgartificialanalysis.ai/models/qwen3-6-35b-a3b
- IFBench64.4benchmark orgartificialanalysis.ai/models/qwen3-6-35b-a3b
- LiveCodeBench v680.4labhuggingface.co/Qwen/Qwen3.6-35B-A3B
- AIME 202692.7labhuggingface.co/Qwen/Qwen3.6-35B-A3B
- MATH-50084.8labhuggingface.co/RedHatAI/Qwen3.6-35B-A3B-FP8
- MMLU-Redux 2.093.3labhuggingface.co/Qwen/Qwen3.6-35B-A3B
- IFEval93.09labhuggingface.co/RedHatAI/Qwen3.6-35B-A3B-FP8
- HumanEval+92.8labhuggingface.co/Aleph-Alpha/Kolibri-1
- AA-Omniscience-22.2benchmark orgartificialanalysis.ai/models/qwen3-6-35b-a3b
- AA-LCR v1.171.7benchmark orgartificialanalysis.ai/models/qwen3-6-35b-a3b
Open weights Qwen/Qwen3.6-35B-A3B MoE (HF repo 2026-04-15), thinking mode (default). GPQA/IFBench/AA-LCR/Omniscience from AA (card GPQA 86.0). MMLU-Redux, LiveCodeBench v6 and AIME 2026 (full I+II) from the model card. MATH-500 84.80 and IFEval 93.09 (metric level not stated) are the BF16 baseline column on RedHat's FP8 card; Liquid's LFM2.5 card reports MATH-500 96.73 for the same model, a large conflict. HumanEval+ 92.8 is Aleph Alpha's baseline column on the Kolibri-1 card. Kolibri also has MMLU-ProX DE 81.9 (full ProX, not Lite), not included.
Qwen3.6-27B
- GPQA Diamond87.8labhuggingface.co/Qwen/Qwen3.6-27B
- IFBench67.55benchmark orgartificialanalysis.ai/models/qwen3-6-27b
- LiveCodeBench v683.9labhuggingface.co/Qwen/Qwen3.6-27B
- AIME 202694.1labhuggingface.co/Qwen/Qwen3.6-27B
- MMLU-Redux 2.093.5labhuggingface.co/Qwen/Qwen3.6-27B
- AA-Omniscience-20.02benchmark orgartificialanalysis.ai/models/qwen3-6-27b
- AA-LCR v1.177.33benchmark orgartificialanalysis.ai/models/qwen3-6-27b
Open weights Qwen/Qwen3.6-27B (HF repo 2026-04-21), thinking mode. GPQA, MMLU-Redux, LiveCodeBench v6 and AIME 2026 (full I+II) from the model card (AA GPQA 84.24, Epoch 85.9). IFBench, AA-LCR and Omniscience from AA's 'Qwen3.6 27B (Reasoning)' page; the Qwen3.8-27B card has IFBench 69.1 for it.
Gemma 4 31B
- GPQA Diamond84.3labhuggingface.co/google/gemma-4-31B-it
- IFBench76lab paperarxiv.org/abs/2607.02770
- LiveCodeBench v680labhuggingface.co/google/gemma-4-31B-it
- AIME 202689.2labhuggingface.co/google/gemma-4-31B-it
- MATH-50087.93communityhuggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic
- MMLU-Redux 2.093.7labhuggingface.co/Qwen/Qwen3.6-35B-A3B
- IFEval94.58communityhuggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic
- Global-MMLU-Lite de91.75benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german
- Global-MMLU-Lite en90.5benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english
- AA-Omniscience-47.9benchmark orgartificialanalysis.ai/models/gemma-4-31b
- AA-LCR v1.169.7benchmark orgartificialanalysis.ai/models/gemma-4-31b
MATH-500 and IFEval: BF16 baseline rows (google/gemma-4-31B-it, thinking on, 3 seeds) on RedHat's FP8 card; thinking off they are 89.40 / 90.70. The tech report's IFEval 98.9 (thinking) doesn't state the metric, so it is not used. MMLU-Redux 93.7 (thinking) is Alibaba's baseline column on the Qwen3.6-35B-A3B and Qwen3.6-27B cards. Global-MMLU-Lite from Kaggle (de ±2.70, en ±2.87). AA-LCR/Omniscience are AA's current values. Our earlier comparison's community HumanEval+ 91.5 has no recorded source and is left out.
Gemma 4 26B A4B
- GPQA Diamond79.2benchmark orgartificialanalysis.ai/models/gemma-4-26b-a4b
- IFBench72.4benchmark orgartificialanalysis.ai/models/gemma-4-26b-a4b
- LiveCodeBench v677.1labhuggingface.co/google/gemma-4-26B-A4B-it
- AIME 202688.3labhuggingface.co/google/gemma-4-26B-A4B-it
- IFEval98.5labarxiv.org/abs/2607.02770
- Global-MMLU-Lite de85.96benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german
- Global-MMLU-Lite en87.97benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english
- AA-Omniscience-50.8benchmark orgartificialanalysis.ai/models/gemma-4-26b-a4b
- AA-LCR v1.165.7benchmark orgartificialanalysis.ai/models/gemma-4-26b-a4b
Open weights google/gemma-4-26B-A4B-it (released 2026-03-31/04-02), thinking on. GPQA/IFBench/AA-LCR/Omniscience from AA (card GPQA 82.3; tech report IFBench 72.0). AIME 2026 and LiveCodeBench v6 from the model card; IFEval 98.5 from the tech report Table 5 only, prompt/instruction level and strict/loose not stated. Global-MMLU-Lite from Kaggle (de ±3.41, en ±3.19), default API settings.
Nemotron-3.5-Lightning-30B
- GPQA Diamond75.44labhuggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
- IFBench71.88labhuggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
- MATH-50096.4labhuggingface.co/LiquidAI/LFM2.5-24B-A2B
- IFEval94.19labhuggingface.co/LiquidAI/LFM2.5-24B-A2B
- AA-Omniscience-17.7benchmark orgartificialanalysis.ai/models/nemotron-3-5-lightning
- AA-LCR v1.160.3benchmark orgartificialanalysis.ai/models/nemotron-3-5-lightning
GPQA and IFBench (loose, not prompt-level strict) from NVIDIA's BF16 card; tech report Table 22 has GPQA 76.45, IFBench 72.00 loose, AA-LCR 52.38. AA-LCR/Omniscience from AA. MATH-500 96.40 and IFEval 94.19 (metric level not stated) are Liquid's baseline column on the LFM2.5-24B-A2B card; our earlier comparison's unsourced community numbers (MATH-500 90.2, IFEval 87.8, HumanEval+ 81.1) are left out. No LiveCodeBench v6, AIME 2026, MultiChallenge or Global-MMLU-Lite found.
Nemotron 3 Nano 30B-A3B
- GPQA Diamond75.7benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning
- IFBench71.1benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning
- LiveCodeBench v668.3labhuggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
- AIME 202689.9labhuggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B
- HumanEval+92.4labhuggingface.co/Aleph-Alpha/Kolibri-1
- AA-Omniscience-51.6benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning
- AA-LCR v1.138benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning
- MultiChallenge38.5labhuggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
Open weights nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, reasoning. GPQA/IFBench/AA-LCR/Omniscience from AA (card: GPQA 73.0, IFBench prompt 71.5, AA-LCR 35.9). LiveCodeBench v6 (2408-2505 window) and MultiChallenge from its own card. AIME 2026 89.9 is NVIDIA's baseline column on the Nemotron-Cascade-2 card; HumanEval+ 92.4 is Aleph Alpha's baseline column on the Kolibri-1 card (Kolibri also: LCB v6 71.3, AIME 2026 87.9). Card MMLU-ProX 59.5 and WMT24++ 86.2 are multi-language averages, not included.
Nemotron Cascade 2 30B-A3B
- GPQA Diamond75.8benchmark orgartificialanalysis.ai/models/nemotron-cascade-2-30b-a3b
- IFBench80.4benchmark orgartificialanalysis.ai/models/nemotron-cascade-2-30b-a3b
- LiveCodeBench v687.2labhuggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B
- AIME 202690.9labhuggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B
- MMLU-Redux 2.086.3labhuggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B
- AA-Omniscience-51.5benchmark orgartificialanalysis.ai/models/nemotron-cascade-2-30b-a3b
- AA-LCR v1.139.3benchmark orgartificialanalysis.ai/models/nemotron-cascade-2-30b-a3b
- MultiChallenge45.3labhuggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B
Open weights nvidia/Nemotron-Cascade-2-30B-A3B. GPQA/IFBench/AA-LCR/Omniscience from AA (card: GPQA 76.1, IFBench prompt 82.9, AA-LCR 39.1). LiveCodeBench v6 and AIME 2026 without tool-integrated reasoning (with TIR: 88.4 / 95.0); MMLU-Redux version not stated. Card MMLU-ProX 72.5 and WMT24++ 84.1 are multi-language averages, not included.
Muse Glimmer 30B
- GPQA Diamond83.5labhuggingface.co/meta-models/Muse-Glimmer-30B
- IFBench77labhuggingface.co/meta-models/Muse-Glimmer-30B
- AIME 202694.7labhuggingface.co/meta-models/Muse-Glimmer-30B
- AA-Omniscience-32.9benchmark orgartificialanalysis.ai/models/muse-glimmer
- AA-LCR v1.183.3benchmark orgartificialanalysis.ai/models/muse-glimmer
Carried over from our earlier comparison; AA-LCR/Omniscience are AA's current values. Only 5 benchmarks found (NVIDIA's NVFP4 card has IFBench strict 76.58, GPQA 83.81, AA-LCR 76.25).
Ornith-1.5 35B
- GPQA Diamond89.2labhuggingface.co/ornith-ai/Ornith-1.5-35B-A3B
- HumanEval+92.7communityhuggingface.co/fetchingcat/occamy-ornith-35b-a3b-mtp-gguf
Only 2 sourced benchmarks: GPQA from the Ornith card, HumanEval+ 92.7 from a community GGUF card, run by that third party on the Q4_K_M quant, not the full-precision weights. Our earlier comparison's MATH-500 88.2 / IFEval 77.3 have no recorded source and are left out. Not on AA.
Open-weight, above 35B 18 models
Qwen3.8-Max
- GPQA Diamond92.6labhuggingface.co/Qwen/Qwen3.8-2.4T-A95B
- IFBench82.8labhuggingface.co/Qwen/Qwen3.8-2.4T-A95B
- AA-Omniscience11.98benchmark orgartificialanalysis.ai/models/qwen3-8-max
- AA-LCR v1.180.33benchmark orgartificialanalysis.ai/models/qwen3-8-max
Qwen3.8-Max: Alibaba Qwen flagship, 2.4T-total / 95B-active MoE (Gated DeltaNet hybrid), released 2026-08-03 (blog https://qwen.ai/blog?id=qwen3.8; MathArena lists 2026-08-02). API model id 'qwen3.8-max' (Qwen Cloud; OpenRouter 'qwen/qwen3.8-max'); $2/$6 per M in/out. Hybrid thinking; thinking on by default; reasoning_effort xhigh (default)/medium/low. Open weights exist: Qwen/Qwen3.8-2.4T-A95B (HF, ~2026-08-12, 'qwen3.8-max' custom license); the card says Qwen3.8-Max is the official API version built on that checkpoint, adding vision, non-thinking mode, 1M context and built-in tools. The open checkpoint is thinking-only. Artificial Analysis tracks two API snapshots: 'Qwen3.8 Max' (0803, now deprecated) and 'Qwen3.8 Max (0902)', released 2026-09-02, which is the current one on the /models/qwen3-8-max page. AA lists both as proprietary and reasoning. It also tracks the open 2.4T-A95B checkpoint separately. Official lab numbers come from the launch table (Qwen3.8-Max column, thinking mode, presumably default xhigh effort). The AA numbers here are from the 0902 snapshot. GPQA/IFBench are the official launch table (0803 snapshot; IFBench strict/loose not stated). AA-Omniscience and AA-LCR are the AA 0902 snapshot (0803: 3.4 / 78.33). Carried over from our earlier comparison; not re-checked against the URLs.
Qwen3.5-397B-A17B
- GPQA Diamond89.3benchmark orgartificialanalysis.ai/models/qwen3-5-397b-a17b
- IFBench78.8benchmark orgartificialanalysis.ai/models/qwen3-5-397b-a17b
- LiveCodeBench v683.6labhuggingface.co/Qwen/Qwen3.5-397B-A17B
- AIME 202693.3labhuggingface.co/Qwen/Qwen3.6-27B
- MMLU-Redux 2.094.9labhuggingface.co/Qwen/Qwen3.5-397B-A17B
- IFEval92.6labhuggingface.co/Qwen/Qwen3.5-397B-A17B
- AA-Omniscience-30.8benchmark orgartificialanalysis.ai/models/qwen3-5-397b-a17b
- AA-LCR v1.177.3benchmark orgartificialanalysis.ai/models/qwen3-5-397b-a17b
- MultiChallenge67.6labhuggingface.co/Qwen/Qwen3.5-397B-A17B
Open weights Qwen/Qwen3.5-397B-A17B (HF repo 2026-02-16), thinking mode (default), effort not stated. GPQA/IFBench/AA-LCR/Omniscience from AA (card self-reports GPQA 88.4, IFBench 76.5, AA-LCR 68.7). MMLU-Redux, IFEval, MultiChallenge and LiveCodeBench v6 from the model card; IFEval/IFBench strict vs loose not stated. AIME 2026 93.3 is the full AIME 2026 I+II as quoted in the Qwen3.6 cards; the 3.5 card's own AIME26 91.3 appears to be a partial set. Card also has MMLU-ProX 84.7 (29-language average) and WMT24++ 78.9 (55-language XCOMET-XXL average), not comparable, not included.
Qwen3.5-122B-A10B
- GPQA Diamond85.7benchmark orgartificialanalysis.ai/models/qwen3-5-122b-a10b
- IFBench75.7benchmark orgartificialanalysis.ai/models/qwen3-5-122b-a10b
- LiveCodeBench v678.9labhuggingface.co/Qwen/Qwen3.5-122B-A10B
- MMLU-Redux 2.094labhuggingface.co/Qwen/Qwen3.5-122B-A10B
- IFEval93.4labhuggingface.co/Qwen/Qwen3.5-122B-A10B
- AA-Omniscience-41.5benchmark orgartificialanalysis.ai/models/qwen3-5-122b-a10b
- AA-LCR v1.176.3benchmark orgartificialanalysis.ai/models/qwen3-5-122b-a10b
- MultiChallenge61.5labhuggingface.co/Qwen/Qwen3.5-122B-A10B
Open weights Qwen/Qwen3.5-122B-A10B (HF repo 2026-02-24), thinking mode (default). GPQA/IFBench/AA-LCR/Omniscience from AA (card: GPQA 86.6, IFBench 76.1, AA-LCR 66.9). Others from the model card; IFEval strict vs loose not stated. Card's MMLU-ProX 82.2 / WMT24++ 78.3 are multi-language averages, not included.
Kimi K3
- GPQA Diamond93.5labhuggingface.co/moonshotai/Kimi-K3
- IFBench74.4benchmark orghuggingface.co/nvidia/Kimi-K3-NVFP4
- LiveCodeBench v687.19benchmark orgvals.ai/benchmarks/lcb
- AIME 202696.67benchmark orgmatharena.ai/models/moonshot_k3
- MATH-50090labhuggingface.co/RedHatAI/Kimi-K3-INT4
- AA-Omniscience19.7benchmark orgartificialanalysis.ai/models/kimi-k3
- AA-LCR v1.188.67benchmark orgartificialanalysis.ai/models/kimi-k3
Open weights moonshotai/Kimi-K3, released 2026-07-16, effort max. As in our earlier comparison, plus IFBench 74.4: NVIDIA's rerun of the original checkpoint (baseline row of the NVFP4 card), not Moonshot's number. AA-LCR: the card quotes 74.7 (AA, 2026-07-23); AA now shows 88.67. MATH-500 90 (reported as a whole-number percentage) is RedHat's 'preliminary smoke-test' run of the original checkpoint (RedHatAI/Kimi-K3-INT4 card).
GLM-5.3
- GPQA Diamond91.7benchmark orgartificialanalysis.ai/models/glm-5-3
- IFBench65.33labhuggingface.co/nvidia/GLM-5.3-NVFP4
- LiveCodeBench v680.53benchmark orgvals.ai/benchmarks/lcb
- MATH-50095.07labhuggingface.co/RedHatAI/GLM-5.3-NVFP4
- IFEval89.77labhuggingface.co/RedHatAI/GLM-5.3-NVFP4
- AA-Omniscience14.3benchmark orgartificialanalysis.ai/models/glm-5-3
- AA-LCR v1.179.7benchmark orgartificialanalysis.ai/models/glm-5-3
Open weights zai-org/GLM-5.3 (2026-08-14), effort max. GPQA/AA-LCR/Omniscience from AA (Vals GPQA 88.13, Epoch 90.9). LiveCodeBench v6 from Vals. IFBench 65.33 is NVIDIA's rerun of the FP8 original (NVFP4 card; same row has LCB 52.95, far below Vals). IFEval 89.77 (prompt strict) and MATH-500 95.07 are RedHat's rerun of the FP8 original (RedHatAI/GLM-5.3-NVFP4 card). Z.ai's own card reports only agentic/cyber/HLE.
GLM-5.3-Flash
- GPQA Diamond91.2benchmark orgartificialanalysis.ai/models/glm-5-3-flash
- IFBench61.3labhuggingface.co/nvidia/GLM-5.3-Flash-NVFP4
- LiveCodeBench v680.51benchmark orgvals.ai/benchmarks/lcb
- AA-Omniscience7.5benchmark orgartificialanalysis.ai/models/glm-5-3-flash
- AA-LCR v1.180benchmark orgartificialanalysis.ai/models/glm-5-3-flash
Open weights zai-org/GLM-5.3-Flash (2026-08-26), effort max. GPQA/AA-LCR/Omniscience from AA (Vals GPQA 86.36, Epoch 90.15). LiveCodeBench v6 from Vals. IFBench 61.3 is NVIDIA's rerun (BF16 row of its NVFP4 card); NVIDIA's Nemotron 3.5 Super VL card has a different run (IFBench 57.7, LCB 80.6, AA-LCR 79.5, Omniscience 14.3) plus MMLU-ProX 83.0 / WMT24++ 86.13 multi-language averages, not included.
K-EXAONE 2.0
- GPQA Diamond82.9benchmark orgartificialanalysis.ai/models/k-exaone-2-0-0803
- IFBench72.6labhuggingface.co/papers/2608.04505
- AIME 202692.3labhuggingface.co/papers/2608.04505
- IFEval92.4labhuggingface.co/papers/2608.04505
- Global-MMLU-Lite de88.3labhuggingface.co/papers/2608.04505
- AA-Omniscience-6.6benchmark orgartificialanalysis.ai/models/k-exaone-2-0-0803
- AA-LCR v1.160benchmark orgartificialanalysis.ai/models/k-exaone-2-0-0803
Open weights LGAI-EXAONE/K-EXAONE-2.0-750B-A37B (HF repo 2026-07-29), reasoning mode. GPQA/AA-LCR/Omniscience from AA (card GPQA 82.2, AA-LCR 56.2). AIME 2026, IFEval, IFBench from the card / tech report Table 6 (strict vs loose not stated); Global-MMLU-Lite de 88.3 from tech report Table 8 (per-language; no English number, the headline 86.6 is a 9-language non-English average). Tech report Table 9 also has WMT24++ en->de 85.1 with the metric unstated, not included.
Solar Open2 250B
- GPQA Diamond85.7benchmark orgartificialanalysis.ai/models/solar-open2-250b
- IFBench80labhuggingface.co/upstage/Solar-Open2-250B
- LiveCodeBench v692.4labhuggingface.co/upstage/Solar-Open2-250B
- AIME 202695.7labhuggingface.co/upstage/Solar-Open2-250B
- AA-Omniscience-1.8benchmark orgartificialanalysis.ai/models/solar-open2-250b
- AA-LCR v1.171.7benchmark orgartificialanalysis.ai/models/solar-open2-250b
- MultiChallenge61labhuggingface.co/upstage/Solar-Open2-250B
Open weights upstage/Solar-Open2-250B (HF repo 2026-07-22; tech report arXiv 2607.20062). GPQA/AA-LCR/Omniscience from AA (card GPQA 86.3, AA-LCR 62.3). IFBench (strict vs loose not stated), LiveCodeBench v6, AIME 2026 (MathArena set) and MultiChallenge from the model card; card recommends effort high, the table doesn't state it.
Solar Open 100B
- GPQA Diamond65.7benchmark orgartificialanalysis.ai/models/solar-open-100b-reasoning
- IFBench57.7benchmark orgartificialanalysis.ai/models/solar-open-100b-reasoning
- LiveCodeBench v656.5labhuggingface.co/upstage/Solar-Open2-250B
- AIME 202687.7labhuggingface.co/upstage/Solar-Open2-250B
- AA-Omniscience-54.5benchmark orgartificialanalysis.ai/models/solar-open-100b-reasoning
- AA-LCR v1.141benchmark orgartificialanalysis.ai/models/solar-open-100b-reasoning
- MultiChallenge40.5labhuggingface.co/upstage/Solar-Open2-250B
Open weights Upstage Solar Open 100B (reasoning). GPQA/IFBench/AA-LCR/Omniscience from AA. LiveCodeBench v6, AIME 2026 and MultiChallenge from the baseline column of the Solar-Open2-250B model card (Upstage's harness).
Command A+
- GPQA Diamond76.1benchmark orgartificialanalysis.ai/models/command-a-plus
- IFBench73.9benchmark orgartificialanalysis.ai/models/command-a-plus
- LiveCodeBench v686.1labhuggingface.co/upstage/Solar-Open2-250B
- AIME 202696labhuggingface.co/upstage/Solar-Open2-250B
- AA-Omniscience-4benchmark orgartificialanalysis.ai/models/command-a-plus
- AA-LCR v1.152.7benchmark orgartificialanalysis.ai/models/command-a-plus
- MultiChallenge45.8labhuggingface.co/upstage/Solar-Open2-250B
Cohere Command A+ (open weights, non-commercial licence). GPQA/IFBench/AA-LCR/Omniscience from AA. LiveCodeBench v6, AIME 2026 and MultiChallenge are Upstage's runs, from the baseline column of the Solar-Open2-250B model card (Upstage also got GPQA 75.6, IFBench 73.9, AA-LCR 46.0).
gpt-oss-120b
- GPQA Diamond78.2benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- IFBench69benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- LiveCodeBench v683.23benchmark orgvals.ai/benchmarks/lcb
- MATH-50094.8benchmark orgvals.ai/benchmarks/math500
- AA-Omniscience-49.2benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- AA-LCR v1.152benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- MultiChallenge45.34benchmark orglabs.scale.com/leaderboard/multichallenge
Open weights openai/gpt-oss-120b (2025-08-05). AA numbers at high effort (paper Table 3 GPQA high 80.1). MATH-500 and LiveCodeBench from Vals (Fireworks provider, MATH-500 archived 2026-01-09); Kaggle's MATH-500 run gives 76.53, a large conflict. MultiChallenge from Scale.
Nemotron 3 Ultra
- GPQA Diamond86.7benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b
- IFBench81.4benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b
- LiveCodeBench v689labhuggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
- AA-Omniscience-0.4benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b
- AA-LCR v1.179.3benchmark orgartificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b
- MultiChallenge63.8labhuggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
Open weights nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 (2026-06-04), thinking on. GPQA/IFBench/AA-LCR/Omniscience from AA; AA's IFBench is used because the card's 81.7 is prompt-loose (card AA-LCR 65.4). LiveCodeBench v6 and MultiChallenge from the BF16 model card / tech report (Vals LCB 85.98). Card MMLU-ProX 83.0 (10-language avg) and WMT24++ 83.7 (en->xx avg) not included.
Ring-2.6-1T
- GPQA Diamond85.7benchmark orgartificialanalysis.ai/models/ring-2-6-1t
- IFBench44.6benchmark orgartificialanalysis.ai/models/ring-2-6-1t
- LiveCodeBench v686.95labarxiv.org/pdf/2606.15079
- AIME 202695.78labarxiv.org/pdf/2606.15079
- AA-Omniscience-37.7benchmark orgartificialanalysis.ai/models/ring-2-6-1t
- AA-LCR v1.170benchmark orgartificialanalysis.ai/models/ring-2-6-1t
Open weights inclusionAI/Ring-2.6-1T (HF repo 2026-05-14, MIT). GPQA/IFBench/AA-LCR/Omniscience from AA. AIME 2026 (avg@64) and LiveCodeBench v6 (avg@4, 2408-2505 window) at xhigh from the Ling/Ring 2.6 tech report Table 6 (card text: AIME26 95.83, GPQA 88.27, probably a newer checkpoint).
Inkling
- GPQA Diamond87.2benchmark orgartificialanalysis.ai/models/inkling
- IFBench79.8labthinkingmachines.ai/news/introducing-inkling/
- LiveCodeBench v685.52benchmark orgvals.ai/benchmarks/lcb
- AIME 202697.1labthinkingmachines.ai/news/introducing-inkling/
- AA-Omniscience2benchmark orgartificialanalysis.ai/models/inkling
- AA-LCR v1.177.3benchmark orgartificialanalysis.ai/models/inkling
Open weights thinkingmachines/Inkling, effort 0.99 (xhigh). GPQA/AA-LCR/Omniscience from AA (Vals GPQA 87.12, Epoch 88.26). IFBench (metric variant not stated) and AIME 2026 from Thinking Machines' launch post / HF card. LiveCodeBench v6 from Vals. Its Global-MMLU-Lite 88.7 is an all-language average, not included.
Inkling-Small
- GPQA Diamond89.5labthinkingmachines.ai/news/inkling-small/
- IFBench82.2labthinkingmachines.ai/news/inkling-small/
- LiveCodeBench v685.93benchmark orgvals.ai/benchmarks/lcb
- AIME 202695.5labthinkingmachines.ai/news/inkling-small/
- AA-Omniscience-8.9benchmark orgartificialanalysis.ai/models/inkling-small
- AA-LCR v1.175.7benchmark orgartificialanalysis.ai/models/inkling-small
Open weights thinkingmachines/Inkling-Small, xhigh. GPQA/IFBench/AIME 2026 from Thinking Machines' post; AA-LCR/Omniscience from AA. LiveCodeBench v6 from Vals (NVIDIA's Nemotron 3.5 tech report Table 10 has 88.25, plus IFBench strict 79.31). Card Global-MMLU-Lite 86.7 is an aggregate, not per language; NVIDIA TR MMLU-ProX 83.97 / WMT24++ 84.60 are multi-language averages; not included.
Step 5 Preview
- GPQA Diamond93.5labstepfun.com/step-5-preview
- AA-Omniscience16.4benchmark orgartificialanalysis.ai/models/step-5
- AA-LCR v1.188.3benchmark orgartificialanalysis.ai/models/step-5
Open weights stepfun-ai/Step-5-Preview (weights ~2026-10-15), effort High. Only 3 benchmarks found; StepFun's card lists only agentic benchmarks.
Muse Spark Open
- GPQA Diamond89benchmark orgartificialanalysis.ai/models/muse-spark-open
- AA-Omniscience24.6benchmark orgartificialanalysis.ai/models/muse-spark-open
- AA-LCR v1.178.3benchmark orgartificialanalysis.ai/models/muse-spark-open
Open weights meta-models/Muse-Spark-Open-816B-A42B (2026-10-15, gated), xhigh. Only 3 benchmarks found (Epoch GPQA 88.6; Meta's card: GPQA 89.0, Omniscience 25).
MiMo-V2.6-Pro
- AA-Omniscience8.4benchmark orgartificialanalysis.ai/models/mimo-v2-6-pro
- AA-LCR v1.186.3benchmark orgartificialanalysis.ai/models/mimo-v2-6-pro
Open weights XiaomiMiMo/MiMo-V2.6-Pro-RL (2026-09-21, MIT). Only AA-LCR and Omniscience found; Xiaomi's card and tech report list only agentic/coding benchmarks.
Mistral 2 models
Mistral Large 4 (Preview)
- GPQA Diamond88.1benchmark orgepoch.ai/benchmarks/gpqa-diamond
- AA-Omniscience-5.3benchmark orgartificialanalysis.ai/models/mistral-large-4
- AA-LCR v1.181.33benchmark orgartificialanalysis.ai/models/mistral-large-4
Released 2026-10-06 as an API preview; open weights (mistralai/Mistral-Large-4-1T-A52B) ETA 2026-10-31. GPQA: Epoch run mistral-large-4-0_high. Omniscience/AA-LCR from AA's model page (record 'Mistral Large 4 Preview': Omniscience -5.3, AA-LCR 0.8133). Mistral's launch post has only agentic/coding/cyber/safety benchmarks and promises more with the weights. Checked 2026-10-19.
Mistral Medium 3.5
- GPQA Diamond74.8benchmark orgartificialanalysis.ai/models/mistral-medium-3-5
- IFBench68.8benchmark orgartificialanalysis.ai/models/mistral-medium-3-5
- LiveCodeBench v684.9labhuggingface.co/upstage/Solar-Open2-250B
- AIME 202689labhuggingface.co/upstage/Solar-Open2-250B
- AA-Omniscience-37benchmark orgartificialanalysis.ai/models/mistral-medium-3-5
- AA-LCR v1.169benchmark orgartificialanalysis.ai/models/mistral-medium-3-5
- MultiChallenge49.8labhuggingface.co/upstage/Solar-Open2-250B
GPQA/IFBench/Omniscience/LCR from AA; LiveCodeBench v6, AIME 2026 and MultiChallenge from Upstage's harness (which got GPQA 77.5 / IFBench 69.0). Carried over from our earlier comparison; the Upstage numbers are coded L (another lab's card that uses the model as a baseline), as for other models. Not re-checked against the URLs.
OpenAI 5 models
GPT-6.1 Sol
- GPQA Diamond95.39benchmark orgepoch.ai/benchmarks/gpqa-diamond
- MATH-50099.01benchmark orgkaggle.com/benchmarks/open-benchmarks/math-500
- Global-MMLU-Lite de85.25benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german
- Global-MMLU-Lite en91benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english
- AA-Omniscience41.53benchmark orgartificialanalysis.ai/models/gpt-6-1-sol
- AA-LCR v1.183benchmark orgartificialanalysis.ai/models/gpt-6-1-sol
- MultiChallenge81.95benchmark orglabs.scale.com/leaderboard/multichallenge
Released 2026-09-29; the current Sol tier, successor to GPT-6 Sol. GPQA: Epoch run gpt-6.1-sol_max, 95.39 ±1.38. MATH-500 99.01 ±0.22, Global-MMLU-Lite de 85.25 ±3.48 and en 91.0 ±2.80: Kaggle, evaluated 2026-10-05, default API settings. MultiChallenge 81.95 ±4.16: Scale leaderboard (official judge). AA numbers at max effort. Checked 2026-10-19.
GPT-6 Sol
- GPQA Diamond94.32benchmark orgepoch.ai/benchmarks/gpqa-diamond
- MATH-50098.64benchmark orgkaggle.com/benchmarks/open-benchmarks/math-500
- AA-Omniscience27.12benchmark orgartificialanalysis.ai/models/gpt-6-sol
- AA-LCR v1.183.67benchmark orgartificialanalysis.ai/models/gpt-6-sol
Released 2026-09-22; superseded by GPT-6.1 Sol on 2026-09-29 (AA marks it deprecated). GPQA: Epoch run gpt-6-sol_max, 94.32 ±1.49. MATH-500: Kaggle, 98.64 ±0.25, evaluated 2026-10-01, default API settings. AA numbers at max effort. OpenAI's launch post reports only agentic benchmarks (AutomationBench, DeepSWE, OSWorld 2.0). Kaggle also has AIME 2025 100 and MMLU-Pro 88.1 (not in our set). Checked 2026-10-19.
GPT-5.6 Sol
- GPQA Diamond94.6labopenai.com/index/gpt-5-6/
- IFBench72.7benchmark orgartificialanalysis.ai/models/gpt-5-6-sol
- LiveCodeBench v682.6benchmark orgvals.ai/models/openai_gpt-5.6-sol
- AIME 202699.9benchmark orgthinkingmachines.ai/news/introducing-inkling/
- MATH-50096.7benchmark orgkaggle.com/benchmarks/open-benchmarks/math-500
- Global-MMLU-Lite de95.3benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german
- Global-MMLU-Lite en96.3benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english
- AA-Omniscience22benchmark orgartificialanalysis.ai/models/gpt-5-6-sol
- AA-LCR v1.184benchmark orgartificialanalysis.ai/models/gpt-5-6-sol
GPT-5.6 Sol is the flagship tier of the GPT-5.6 family (Sol / Terra / Luna). It is a separate model from GPT-5.6 Luna, the cheapest tier: launch prices were $5/$30 per 1M tokens for Sol vs $1/$6 for Luna. Limited preview June 26, 2026; GA July 9, 2026; API id gpt-5.6-sol. Effort levels: low, medium, high, xhigh, max, plus ultra (multi-agent). The GPQA value is from OpenAI's launch table, which does not state the effort. AA, Vals and the Inkling numbers are at max effort; the Inkling blog says 'max/xhigh' and its HF card says 'xhigh'. Kaggle rows run the default API settings, effort not stated. Other GPQA numbers: AA max 94.1, Vals max 95.2, Epoch max 93.5. Kimi K3 card reports AA-LCR 73.7 for Sol (max), vs AA's 84. Carried over from our earlier comparison; not re-checked against the URLs.
GPT-5.6 Luna
- GPQA Diamond92.3labopenai.com/index/gpt-5-6/
- IFBench67.3benchmark orgthinkingmachines.ai/news/inkling-small/
- AIME 202697.6benchmark orgthinkingmachines.ai/news/inkling-small/
- MATH-50093.7benchmark orgkaggle.com/benchmarks/open-benchmarks/math-500
- Global-MMLU-Lite de84.5benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german
- Global-MMLU-Lite en88.3benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english
- AA-Omniscience-10benchmark orgartificialanalysis.ai/models/gpt-5-6-luna
- AA-LCR v1.184benchmark orgartificialanalysis.ai/models/gpt-5-6-luna
AA numbers at max effort. IFBench and AIME 2026 from Thinking Machines Lab (effort 0.99; its GPQA there was 89.5). Carried over from our earlier comparison; not re-checked against the URLs.
gpt-oss-120b
- GPQA Diamond78.2benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- IFBench69benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- LiveCodeBench v683.23benchmark orgvals.ai/benchmarks/lcb
- MATH-50094.8benchmark orgvals.ai/benchmarks/math500
- AA-Omniscience-49.2benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- AA-LCR v1.152benchmark orgartificialanalysis.ai/models/gpt-oss-120b
- MultiChallenge45.34benchmark orglabs.scale.com/leaderboard/multichallenge
Open weights openai/gpt-oss-120b (2025-08-05). AA numbers at high effort (paper Table 3 GPQA high 80.1). MATH-500 and LiveCodeBench from Vals (Fireworks provider, MATH-500 archived 2026-01-09); Kaggle's MATH-500 run gives 76.53, a large conflict. MultiChallenge from Scale.
Anthropic 3 models
Claude Opus 5.5
- GPQA Diamond90.59benchmark orgepoch.ai/benchmarks/gpqa-diamond
- HumanEval+95.1communityisclaudedumb.today/
- AA-Omniscience46.42benchmark orgartificialanalysis.ai/models/claude-opus-5-5
- AA-LCR v1.184.67benchmark orgartificialanalysis.ai/models/claude-opus-5-5
Released 2026-09-22. All at max effort. GPQA: Epoch run claude-opus-5-5_max, 90.59 ±1.94 (Epoch had Opus 5 at 93.9). AA numbers from the 'max with fallback' run (Intelligence Index v4.3.2). HumanEval+ 95.1: isclaudedumb.today median of 65 daily runs 2026-09-22..2026-10-19 (range 92.6-96.3), run through the Claude Code CLI, so not strict single-generation pass@1; measured the same way as the Opus 5 entry. Anthropic's launch post reports only agentic/coding benchmarks; the system card has Global-MMLU as a 42-language average (94.3), not Global-MMLU-Lite de/en. Not on Kaggle, Scale MultiChallenge or Vals LiveCodeBench; AA no longer runs GPQA/IFBench on new models. Checked 2026-10-19.
Claude Opus 5
- GPQA Diamond93.9benchmark orgepoch.ai/benchmarks/gpqa-diamond
- LiveCodeBench v689benchmark orgvals.ai/benchmarks/lcb
- MATH-50097.6benchmark orgkaggle.com/benchmarks/open-benchmarks/math-500
- HumanEval+94.5communityisclaudedumb.today/
- Global-MMLU-Lite de97benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german
- Global-MMLU-Lite en96.5benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english
- AA-Omniscience37benchmark orgartificialanalysis.ai/models/claude-opus-5
- AA-LCR v1.179benchmark orgartificialanalysis.ai/models/claude-opus-5
AA numbers at max effort. HumanEval+ 94.5: isclaudedumb.today daily run 2026-10-01 (median of 132 runs 94.5, range 92.0-95.7), run through the Claude Code CLI with ~4 turns per task, so not strict single-generation pass@1. IFEval 71.3 left out: no recorded source. Carried over from our earlier comparison; not re-checked against the URLs.
Claude Haiku 5.5
- GPQA Diamond89.6benchmark orgepoch.ai/benchmarks/gpqa-diamond
- MATH-50098.71benchmark orgkaggle.com/benchmarks/open-benchmarks/math-500
- Global-MMLU-Lite de89.75benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-german
- Global-MMLU-Lite en91.25benchmark orgkaggle.com/benchmarks/cohere-labs/global-mmlu-lite-english
- AA-Omniscience10.7benchmark orgartificialanalysis.ai/models/claude-haiku-5-5
- AA-LCR v1.182.7benchmark orgartificialanalysis.ai/models/claude-haiku-5-5
- MultiChallenge67.67benchmark orglabs.scale.com/leaderboard/multichallenge
Released 2026-10-07. GPQA: Epoch run claude-haiku-5-5_max (Kaggle zero-shot GPQA Diamond: 85.86). MATH-500 (±0.25, evaluated 2026-10-09) and Global-MMLU-Lite from Kaggle, default API settings. MultiChallenge from Scale (±5.34). AA-LCR and Omniscience from AA at max effort (xhigh: 78.3 / 6.05). The launch post has none of these benchmarks; the system card has only a 42-language Global-MMLU average (87.8). Not tracked by isclaudedumb.today.
