Last month I wrote an article testing six Claude Code models. This time I’m broadening the scope: my local CPA (local model gateway) has 80 models connected, and I did two things — first, sent real requests to each one to check liveness, then gave the survivors a puzzle-resistant intelligence test, and finally cross-validated against public leaderboards.

Bottom line up front: listing a model ≠ it works. Of the 80 models, 59 are text models, and only about 30 could actually hold a conversation; even fewer passed novel reasoning puzzles. Here’s the full process.

I. Liveness Check: Of 80 Models, First Ask “Can It Talk?”

The method was simple: send each model a message saying “Reply with exactly: OK”, and consider it alive if it returns sensible content. One pitfall worth mentioning —

The entire grok series was misjudged as dead on the first round. The probe responses looked empty, but upon re-examination, they were streaming via SSE, with content nested inside delta.content. My parser wasn’t unwrapping the stream. After fixing that, most of the grok-4.5, grok-4.3 full line, and grok-4.20 models came back to life. Lesson learned: get your response parser right before deciding a model is dead.

The truly dead models expired in various ways:

II. Intelligence Test: Classic Puzzles Can’t Tell the Difference — They’re All in the Training Data

From the survivors, I picked 15 top models and ran a round of 7 classic puzzles (number sequences, syllogisms, chicken-and-rabbit-style word problems). Result: almost everyone scored full marks — zero discrimination.

I switched to harder classics (Python closure traps, trailing zeros of 100!, clock hand overlaps, the 25-horse race problem) and retested: still everyone got 5/6 — the only “wrong” answer was because my grading regex was buggy. Every one of these questions exists in training data; the models have all seen them before.

The method that actually worked: generate unseen Python code on the spot, run it for real in a sandbox to get the ground-truth answer, then have the model predict the output blind. Six code snippets covering dict defaults, slice expansion, memoized recursion, sliding window, reduce, and object sorting. There’s no way to memorize these — it all comes down to genuine execution-based reasoning.

The results finally showed real separation:

Three surprises:

• The agnes family is wildly polarized. pro-alpha scored full marks, while the flash variants only managed 2/6 — the gap within the same family is abnormally large. Don’t rely on agnes flash for anything serious.
• qwen3-coder-next crashed and burned. Wrong answer in 1 second, and two runs produced different answers (15 vs. 14). It’s genuinely fast, but novel reasoning is unstable — fine for mundane grunt work, don’t let it make critical calls.
• grok-4.3-high scored full marks in 11 seconds. The grok family was first confirmed as actually smart; previously it had been dragged down by pipeline issues.

III. Cross-Reference with Public Leaderboard Scores (with credibility notes)

Beyond my own tests, I looked up each model’s public scores online. Important caveat upfront: most of these are vendor-reported, and the degree of third-party verification varies.

IV. Three Traps in Reading Benchmarks

The pitfalls I hit in this round left a deeper impression than reading ten leaderboard articles:

Trap 1: Training data contamination. 25 horses, clock hands, trailing zeros of 100! — every model got these right because they’ve all memorized them. To measure genuine reasoning, either use on-the-fly generated questions or turn to rolling leaderboards like LiveCodeBench.

Trap 2: Vendor-reported ≠ independently verified. qwen3.8-max and the kimi family both face public质疑 about score mismatches; deepseek is the most audit-resistant because of open-source weights. When you see a score, ask first: who ran the test? Can it be reproduced?

Trap 3: Leaderboard scores ≠ your pipeline’s performance. grok-4.6 might look stellar on paper, but a 404 from the upstream gives it a hard zero. qwen3-coder-next has decent coding scores on the leaderboard, but produced inconsistent answers on novel problems across two runs. Ultimately, you have to test on your own machine.

V. Final Ranking (Intelligence × Availability)

Synthesizing “anti-memorization test + public scores + historical baselines”:

1. deepseek-v4-pro-0813 — full marks + hardest leaderboard scores + open-source verifiable
2. qwen3.8-max — full marks + historical baseline IQ 9/9 (代价: slow)
3. glm-5.2-fast-preview — full marks + fastest in the full-score group + Chinese throughput king, the default daily driver
4. qwen3.7-max-preview / kimi-k3 / grok-4.3-high — full-score tier
5. glm-5.2, qwen3.8-27b, deepseek-v4-flash, agnes-2.5-pro-alpha — full marks but each has trade-offs (slow / small capacity / family authenticity doubts)
6. kimi-k2.7-code, grok-4.5, deepseek-v4-flash-0731 — 5/6 tier
7. agnes-flash series, qwen3-coder-next (on novel problems) — not recommended for critical tasks

One-line picking guide: assign high-stakes one-shot tasks to deepseek-0813 or qwen3.8-max, use glm-5.2-fast-preview as your daily workhorse, and keep qwen3-coder-next for dirty grunt sub-agents — but don’t let it make critical judgments.