Home → Guides → GPT-6 Astra vs GPT-6.1 Sol vs Claude Opus 5.5: a coding test with hidden tests
Benchmarks on model cards are run by the people selling the model. So we wrote our own: 13 JavaScript tasks, each answered twice by each model in a single shot, then checked against 286 hidden test cases in a sandbox with no network. Seven tasks are everyday code; six are built around edge cases that are easy to miss.
| Model | Test cases passed | Tasks fully solved | Median time, everyday tasks | Median time, edge-case tasks | Median answer (output tokens) | Average cost per task |
|---|---|---|---|---|---|---|
gpt-6.1-sol | 286/286 (100%) | 26/26 | 12.4 s | 19.8 s | 428 | $0.0027 |
gpt-6-astra | 284/286 (99.3%) | 24/26 | 12.1 s | 17.4 s | 428 | $0.0124 |
claude-opus-5-5 | 275/286 (96.2%) | 21/26 | 4.2 s | 8.9 s | 447 | $0.0056 |
Only gpt-6.1-sol passed everything, in both attempts. gpt-6-astra missed one case — the same one in both attempts. claude-opus-5-5 was two to three times faster than both GPT models and wrote code of similar length, but missed the most cases. Costs are what our gateway billed for these exact requests.
-2^2 is -4, and the other accepted 2 2 as a valid expression. In the everyday calculator one attempt rejected 1 + 2 * 3 because of the spaces, although the task asked for arbitrary whitespace.None of these are exotic: whitespace, a zero-weight edge, a cap on parallel calls. They are exactly the cases that pass a quick manual test and then break in production, which is why the tests check them.
Opus 5.5 answered a typical task in 4–9 seconds. Both GPT models took 12–20 seconds because they reason before they write. For a chat where you watch the answer appear, Opus feels much quicker; for an agent that runs a test suite after each step, the extra ten seconds rarely matters next to the cost of a wrong answer.
Per task, gpt-6.1-sol cost $0.0027, claude-opus-5-5 $0.0056 and gpt-6-astra $0.0124. Astra was about 4.6 times the price of Sol for a slightly lower score. All three wrote answers of almost the same length, so the difference comes from the price per token, not from verbosity.
Every model got the same prompt: the task description plus “Reply with exactly one code block containing the complete CommonJS module”. No system prompt, no examples, no retries. We ran the code in Node 20 inside a container with no network, with a time limit per task; if an answer crashed, each test case was re-run on its own so one crash could not wipe out the rest.
. * + ?, multiplying 2,000-digit numbers without BigInt, an expression evaluator with right-associative ^, a concurrency-limited async pool, counting business days across years, and the lexicographically smallest shortest path.All requests went through our gateway on 12–13 October 2026. Two attempts per task is a small sample, so a difference of a few test cases between two models can come from ordinary run-to-run variation; the zero-failure run of Sol and the repeated Astra bug are not.
gpt-6.1-sol — the only perfect score and the lowest cost.claude-opus-5-5 — answers in a few seconds; pair it with tests, because it is the one most likely to skip a detail in the spec.Current prices: GPT-6.1 Sol · GPT-6 Astra · Claude Opus 5.5.
No subscription, no monthly minimum, no sales call. Top up with USDT and spend what you use — 1 USDT gives you 2 credits of API balance.
Read the 30-second quickstartLanguage: English · 简体中文 · 繁體中文 · 日本語 · 한국어 · Deutsch · Français · Español · Türkçe · Português · Italiano · Bahasa Indonesia · Tiếng Việt · Bahasa Melayu · Polski · Čeština · فارسی · Русский
Prices on this page are filled in automatically from live billing data. Third-party terms are quoted from that party's own documentation.