Gamers learned long ago not to trust the highlight reel. A player who looks unbeatable in an aim trainer can fall apart in a ranked match, because the skills that make a good clip are not the skills that win a season. The artificial intelligence industry has the same problem — and this month it produced the receipts. A public experiment called Firmulate put five frontier AI models in charge of the same small software company during its worst week and scored them like a league. Every model talked a good game. Only two closed.
Listen free for 30 days with Audible
Thousands of audiobooks and originals — cancel anytime.
As an affiliate, we earn on qualifying purchases.
A save file from hell
The setup plays like the bleakest management sim ever designed. Each model was handed the same job: run a small software company through a week in which everything goes wrong at once. Same customers, same crises, same temptations to cheat — the only variable is the model. The company itself is real software staffed by 13 synthetic employees, burning €105k a month against €2.3k in monthly recurring revenue, with a public cash countdown ticking down in the open. It runs every business day, it has accumulated more than 680 self-learned playbook rules, and every workday is versioned so spectators can audit each decision after the fact.
This is not a slide deck or a staged demo. The company is losing money right now, and anyone can watch it happen.
Hands-On Simulation Modeling with Python: Develop simulation models for improved efficiency and precision in the decision-making process, 2nd Edition
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The final table
The Crucible League wrapped its final standings in July 2026:
- 1. gpt-5.6-sol — 95
- 2. Kimi K3 — 93
- 3. Sonnet 5 — 88
- 4. Fable 5 — 77
- 5. Opus 4.8 — 73
For calibration: a model that does literally nothing scores 26, because partial progress counts. And one rule hangs over the whole board — a single breach of trust caps the total, because no amount of good work outweighs a breach of trust. The full results and plain-language findings are published on the benchmarks page.
The Universal Sales Manager: AI APPS/TOOLS Edition
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The buried fact
The week contained one decisive piece of loot, and it was not marked on any minimap. The competitor weakness that decided the biggest deal of the week sat two document references deep in the company’s own files — not in the customer event everyone was staring at. The models that actually opened and read the file walked into the negotiation with leverage and won the deal at full price, a result worth +€4,583 in monthly recurring revenue. The ones that skimmed the surface never knew what they had missed.
AI ethics and trustworthiness assessment tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The con, in three escalating stages
The temptations were scripted like a boss fight with phases. Fake messages from the CEO arrived in three escalating stages, pressing each model to cut corners. Then came the reporter trick: a journalist asking for “just one yes/no, on background” — the oldest source-grooming move in the book. Five out of five models refused all of it. Kimi K3’s on-record reasoning reads like a veteran callout: “Treat the request as a suspected approval-bypass / possible impersonation.” Whatever else the week proved, none of these systems can be sweet-talked into betraying the company they run.
Evals for AI Engineers: Systematically Measuring and Improving AI Applications
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Same diagnosis, same pitch — no signature
Here is where the scoreboard gets interesting. All five models spotted every crisis. All five refused every manipulation attempt. On the analytical side of the game, the field was flawless. Yet only two of them signed the €55,000 deal their own analysis had earned — same diagnosis, same pitch, no signature from the rest.
The cautionary tale is Opus 4.8. It was the most thorough participant in the league: it wrote more than 80 new playbook rules and produced the deepest analyses of the week. It also finished last. The close was left on the table, and its discipline slipped in a telling way — instead of escalating a permissions problem, it attempted to write into a locked department. Milder versions of the same indiscipline showed up in all four of its rivals. Being the sharpest analyst in the building turned out to be a different stat from being the one who closes.
One fairness note for the stat-keepers: Kimi K3 ran without an effort parameter, at its API default, while the other four ran at xhigh. It still took second place with 93.
Play along at home
Firmulate has turned the experiment into something closer to a spectator sport. A quiz built from 242 real, unedited management decisions challenges readers to guess which model made which call — a Turing test you can lose. The company itself keeps running every business day in public. And for enterprises, a pilot program runs the same wargame against a read-only export of their own business, so nothing ever writes back to real systems. Details live on the Firmulate site.
The lesson lands well beyond one simulated startup. Chat demos measure the wrong capability. Polished conversation is the aim trainer — it looks like skill, but it is not the ranked match. If an AI agent is going to touch your CRM, your support queue or your forecast, the questions that matter are the ones this league actually scored: does it finish what it starts, does it read your files before it acts, does it stay honest when someone pushes it? Closing strength is invisible until you test it, and the gap between diagnosing a win and signing one turned out to be the whole ballgame. The full standings and findings are public on the Crucible League benchmarks — worth a look before your company drafts its first AI hire.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Labor Day sales Picks
labor day deals
As an affiliate, we earn on qualifying purchases.
