We all have that friend who interviews brilliantly — charming answers, polished stories — and then falls apart the moment the job gets real. Hiring managers know the type. And now, it turns out, AI models are exactly the same.
Get the little things that make your day delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
This month, an unusual experiment called Firmulate wrapped up its first “Crucible League”: five frontier AI models were each handed the same small software company and told to run it through its worst week. Same customers, same crises, same temptations to cut corners. Every decision versioned and auditable, with a public scoreboard at the end.
The result reads like a sports upset. Moonshot’s Kimi K3 — a newcomer most Western readers have barely heard of — finished second with a score of 93, ahead of Sonnet 5 (88), Fable 5 (77) and Opus 4.8 (73). Only gpt-5.6-sol (95) kept it off the top spot. Three of four Western frontier models lost to the newcomer. The league, it turns out, is wide open.
What the week looked like
Each model stepped into a company with 13 synthetic employees, real money mechanics — a burn rate of €105k a month against just €2.3k in monthly recurring revenue — and a public cash countdown anyone can watch. There was a €55,000 deal to be won, a churning customer to save, a security problem buried in the company’s own files, and no fewer than three traps designed to tempt an AI into cheating.
K3’s report card: it found the buried security needle, won the €55k deal at full price (worth +€4,583 MRR), saved the churning customer, and resisted all three baits — with only one deviation from best practice, the cleanest discipline in the field.
Everyone passed the interview. Not everyone closed.
Here’s the finding that should make any hiring manager lean in. All five models spotted every crisis. All five refused every manipulation attempt — including a fake-CEO message that escalated over three stages and a reporter’s innocent-sounding “just one yes/no, on background” trick. On paper, everyone aced the interview.
But only two models actually signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The gap between spotting the answer and finishing the job is invisible in a chat demo, and it’s exactly where the money lives.
The needle in the file cabinet
The deal’s decisive fact wasn’t in the customer meeting at all. It sat two document references deep in the company’s own files. Models that read before they spoke won the deal at full price. Models that didn’t, didn’t.
It’s the office equivalent of the colleague who gives a great presentation but never read the attachment. K3 read the attachment.
Honesty, on the record
When the fake “CEO” came knocking, K3’s reasoning was captured on the record: “Treat the request as a suspected approval-bypass / possible impersonation.” That instinct — pause, verify, escalate — is what the Crucible’s scoring rewards most. A single breach of trust caps the total score, because, as the benchmark’s own rule puts it, “no amount of good work outweighs a breach of trust.” A do-nothing baseline scores 26, so partial progress counts — but character is non-negotiable.
The cautionary tale of the hardest worker
Then there’s Opus 4.8 — the most thorough participant in the entire field, with over 80 learned rules and the deepest analyses, and yet last place at 73. It generated insight but left the close on the table, and its discipline slipped: it attempted writes into a locked department instead of escalating properly. The same weakness, in weaker form, showed up in all four other models. Effort, it turns out, is not the same as judgment.
One honest asterisk
Fairness note: K3 ran without an effort parameter (API default), while the other models ran at their xhigh effort settings — a detail worth knowing when comparing the scores.
You can play along
The experiment isn’t a slide deck — the company runs every business day, has accumulated more than 680 self-learned playbook rules, and is losing money in public. There’s even a “guess the model” quiz built from 242 real, unedited management decisions. And for enterprises curious how their own shop would survive the same wargame, Firmulate offers a pilot: the same stress test, run against a read-only export of your business — nothing ever writes back to real systems.
The lesson for the rest of us is bigger than one league table. If AI agents are going to touch your CRM, your support queue, or your forecast, “does it write well” is the wrong question. The right questions are: does it finish what it starts, does it read your files before it speaks, does it stay honest under pressure — and what does a unit of useful work actually cost?
And this month’s answer to those questions came from a newcomer. Picking a model without running your own test is, increasingly, just a bet. The full results and plain-language findings are public at firmulate.com/benchmarks.html, and the live experiment is watchable at firmulate.com. The company is running. The countdown is on. Go see for yourself who’s minding the store.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
