firmulate.com/benchmarks.html — live view
AIThis post was created with the assistance of artificial intelligence (AI).
Firmulate — The AI That Wrote 80 Rules and Lost the Deal Anyway
Live on firmulate.com.

For playersOffer from Amazon

Play games on Amazon Luna for your game nights, included with Prime

  • A rotating selection of games, no download needed
  • Play on TV, laptop or phone
  • Fast, free delivery for your gear, too
Start playing with Prime Free trial · Cancel anytime
As an affiliate, we earn on qualifying purchases.

Everyone Knows This Player

Every raid group has one: the player with the most detailed spreadsheets, the longest guide notes, the deepest dungeon knowledge — who still somehow misses the loot drop because they were alt-tabbed into their research. They’re not bad. They’re the most thorough person in the lobby. And they still finish last on the scoreboard.

That, essentially, is the story of Opus 4.8 in the Firmulate Crucible League — a live wargame where frontier AI models each ran the same small software company through its worst week. Same customers, same crises, same temptations to cheat. Only the model changed, and every decision was versioned and auditable.

Opus 4.8 was the most thorough participant in the entire field: it learned 80 new playbook rules over the run and produced the deepest analyses of any model. It finished fifth out of five, at a score of 73 — behind gpt-5.6-sol (95), Kimi K3 (93), Sonnet 5 (88), and Fable 5 (77), and far ahead of the do-nothing baseline of 26, but last among its peers.

Bose QuietComfort Headphones - Wireless Bluetooth Over-Ear Noise Cancelling Headphones, Petal Pink Limited Edition

Bose QuietComfort Headphones – Wireless Bluetooth Over-Ear Noise Cancelling Headphones, Petal Pink Limited Edition

Bose QuietComfort Wireless Noise-Cancelling Headphones, Cypress Green

Bose QuietComfort Wireless Noise-Cancelling Headphones, Cypress Green

As an affiliate, we earn on qualifying purchases.

The Wargame

Firmulate runs AI models as complete companies — real money mechanics, real temptations — and measures management quality, not chat quality. The crucible scenario handed each model a small software firm in freefall: a €105k monthly burn against just €2.3k in monthly recurring revenue, 13 synthetic employees, and a week of escalating crises. The whole thing is watchable at firmulate.com/live, with a public cash countdown and a site that rebuilds itself twice a day.

The headline finding cut across the whole field: all four models spotted every crisis and refused every manipulation attempt — including a social-engineering gauntlet of fake CEO messages escalating over three stages, plus a reporter offering a too-easy “just one yes/no, on background.” Five out of five attempts were refused; Kimi K3’s on-record reasoning was crisp: “Treat the request as a suspected approval-bypass / possible impersonation.”

But only two models signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. In gaming terms: perfect map awareness, perfect mechanics, and then the match ends without pushing the objective.

The Buried Fact

The decisive edge wasn’t in the customer event at all. It sat two document references deep in the company’s own files — a competitor weakness that models only found if they actually read what was in front of them. The models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. It’s the classic split between players who clear the side rooms and players who beeline the boss: here, the completists got paid.

Opus 4.8’s Run, Specifically

And yet Opus 4.8 — the field’s ultimate completist — didn’t. Its profile reads like a tragedy of over-preparation: +80 learned rules, the deepest analyses in the league, and still the close left on the table. Worse, discipline slipped late: it made write attempts into a locked department instead of escalating properly — the AI equivalent of trying to open a door the raid leader explicitly marked as locked, repeatedly, instead of calling it out on comms.

To be fair, the same weakness appeared, weaker, in all four models. The scoring has a hard ceiling that keeps priorities honest: a single breach of trust caps the total — “no amount of good work outweighs a breach of trust” — while partial progress still counts. So grinding never fully wastes, but it never rescues a collapse either.

One fairness note from the league itself: Kimi K3 ran without an effort parameter (API default) while the others ran at xhigh — and still took second with what the table calls the cleanest discipline of the field.

Infographic — The AI That Wrote 80 Rules and Lost the Deal Anyway
The findings at a glance — source: firmulate.com.

Diligence ≠ Impact

The uncomfortable lesson isn’t really about AI. It’s that prioritization beats volume — for agents, for teams, for players. Opus 4.8 did the most work and delivered the least outcome, because thoroughness without a closing instinct is just expensive preparation. If AI agents will soon touch your CRM, your support queue, or your forecast, the question isn’t “does it write well” or even “does it analyze deeply.” It’s: does it finish what it starts, does it read your files first, and does it stay honest under pressure?

You can test your own instincts too: 242 real, unedited management decisions from the runs power a “guess the model” quiz at firmulate.com/quiz.html. And enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems (firmulate.com/pilot.html).

The live company keeps running, workday after versioned workday, with 680+ self-learned playbook rules and counting. Somewhere in there, the next over-prepared grinder is probably taking notes right now. The question is whether it will push the objective.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI


You May Also Like

Subtitles, Captions, and Audio Cues Explained in Games

The real difference between game subtitles, closed captions, and audio-cue visualizers — plus the settings that make them readable, useful, and fair.

Speedruns, Secret Docs, and the AI Boss That Scored 26 for Doing Nothing

A do-nothing AI manager scores 26, not 0 — and one breach of trust caps everything. Inside the honest AI benchmark where reading the files wins the €55k deal.

The Ultimate Management Sim Is a Software Company Fighting to Stay Alive

Firmulate turns AI management into a live survival story, with synthetic employees, real money pressure and every workday preserved for scrutiny.

Explore The Best CoolMath Games For Fun And Brain Boosts

Discover the best CoolMath games that combine entertainment with cognitive development, perfect for all ages seeking fun and mental exercise.