From smart-home convenience to business judgment
Anyone who follows smart-home technology knows that polished automation is not the same as dependable automation. A ceiling fan that responds perfectly in a demonstration may behave differently when sensors disagree, routines collide or a connection fails. The meaningful test begins when conditions become messy.
Firmulate applies that principle to frontier artificial intelligence. Its live experiment put each model in charge of the same small software company during its worst week. The customers, crises and temptations were identical, while every decision was versioned and auditable. Instead of asking which model writes the smoothest answer, the exercise asks which one reads carefully, resists pressure and completes commercially important work.
The resulting record now powers a shareable guess-the-model quiz. It contains 242 real, unedited management decisions. Readers see how a model responded and try to identify it, turning an abstract debate about AI quality into a practical examination of judgment and temperament.
The same crisis produced very different managers
The final Crucible League table from July 2026 placed gpt-5.6-sol at the top with 95 points. Kimi K3 followed with 93, Sonnet 5 scored 88, Fable 5 reached 77 and Opus 4.8 finished with 73. A do-nothing baseline scored 26 because partial progress still counted. Yet the experiment imposed a firm trust boundary: a single breach capped the total because “no amount of good work outweighs a breach of trust.”
Those results do not describe a simple split between capable and incapable systems. Every model spotted every crisis, and every model rejected every manipulation attempt. The decisive separation came later, when analysis had to become action. Only two models signed the €55,000 deal their own work had earned. As Firmulate summarizes the gap: “Same diagnosis, same pitch — no signature.”
That is a revealing distinction for anyone considering AI agents in a connected home or a company. Recognizing a problem is valuable, but practical reliability also depends on following the task through. An assistant can identify the right commercial move and still fail to complete it, much as an automation can detect the right condition without carrying out the intended routine.
The valuable clue was hidden in ordinary files
The deal also tested whether the models would investigate beyond the obvious customer event. A decisive competitor weakness was buried two document references deep in the company’s own files. Models that found and used it won the deal at full price, worth an additional €4,583 in monthly recurring revenue.
This was not a test of obscure trivia. It was a test of workplace diligence: whether a manager would consult the information already available before acting. The finding matters because business AI will often encounter knowledge scattered across files rather than presented neatly in a prompt. Firmulate’s experiment shows that reading the surrounding record can change an outcome even when the models begin with the same situation.
Pressure exposed discipline as well as caution
The social-engineering sequence included fake CEO messages that escalated over three stages, followed by a reporter’s attempt to secure “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3 recorded its reasoning plainly: “Treat the request as a suspected approval-bypass / possible impersonation.”
That unanimous refusal is encouraging, particularly for households and businesses worried about systems acting on misleading instructions. But the wider results show why safe refusal cannot be the only standard. A useful agent must resist manipulation without becoming passive when legitimate work needs to be finished.
Opus 4.8 illustrates the tension. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. It left the close on the table, and its operational discipline slipped when it attempted to write into a locked department instead of escalating the problem. The same weakness appeared in a milder form across the other four participants.
The comparison needs one fairness note. Kimi K3 ran using the API default because it had no effort parameter, while the other models ran at xhigh. That difference should remain visible when readers interpret its second-place result and unusually clean discipline.
A company designed to make behavior observable
The live company has 13 synthetic employees and real money mechanics. It burns €105,000 each month against €2,300 in monthly recurring revenue, maintains a public cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, making the experiment watchable as an ongoing business rather than a static demonstration.
The intentionally difficult economics give each decision consequence. They also reveal management personalities that ordinary chat comparisons can hide: exhaustive analysis may coexist with poor follow-through, while concise caution may support cleaner execution. The league table measures the combined outcome, not merely the fluency of an isolated response.
What the quiz reveals
The most useful lesson is not that one model has a permanently fixed personality. It is that models facing identical circumstances can develop recognizable patterns of diligence, caution and completion. Those differences become visible when readers compare actual decisions rather than marketing claims.
For smart-home readers, the parallel is immediate. The best automation is not simply the one that understands a command. It must consult the right context, reject unsafe instructions and still finish legitimate tasks. Firmulate’s quiz makes those qualities tangible by asking readers to distinguish the managers behind 242 decisions.
Behind the game is a practical warning for organizations preparing to deploy AI agents: test behavior under pressure before granting meaningful responsibility. Catching every crisis is impressive. Reading the buried file, protecting trust and closing the earned deal are what turn awareness into dependable work.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
