The Showroom Demo Problem Comes to Artificial Intelligence
Anyone who has shopped for a smart ceiling fan, a connected thermostat or a whole-home hub knows the ritual. The showroom demo is flawless: lights dim on cue, the fan responds to every voice command, the app glides through its screens. Then you get the thing home, and the real test begins — the firmware update at the worst moment, the sensor that works nine days out of ten, the automation that quietly stops firing. Smart shoppers learned long ago that demo quality and daily reliability are different products.
A live experiment now running in public is making the same point about the AI models that are about to arrive inside your workplace tools — and eventually inside the software that manages your home. Firmulate, an “AI company emulator,” hands frontier AI models the same job: run a small software company through its worst week, and be judged on management quality rather than chat quality. The results should be required reading for anyone who assumes a polished chatbot conversation equals a capable digital worker.
As an affiliate, we earn on qualifying purchases.
One Company, Five Models, One Terrible Week
The setup is deliberately unfair, the way real life is unfair. Each model took over the same small software firm — thirteen synthetic employees, real money mechanics, and a grim financial trajectory: roughly €105,000 a month in burn against just €2,300 in monthly recurring revenue, with a public cash countdown ticking toward zero. Same customers, same crises, same temptations to cheat. Only the model in charge changed. Every decision was versioned and auditable, and the company doesn’t vanish when the test ends — it runs every business day, has already accumulated more than 680 self-learned playbook rules, and can be watched live by anyone.
The Scoreboard
The final Crucible League table, published in July 2026:
- gpt-5.6-sol — 95. Found the buried fact, closed the deal: the complete performance.
- Kimi K3 — 93. The newcomer from Moonshot; one of only two models that signed the deal, while running at default effort as its rivals ran at maximum.
- Sonnet 5 — 88. A strong score that still left the signature unsigned.
- Fable 5 — 77. Found the problems, refused the tricks — but no close.
- Opus 4.8 — 73. The most thorough participant, and the last-place finisher.
For scale: a do-nothing baseline scores 26, because partial progress counts. And one rule towers over the arithmetic — a single breach of trust caps the total. In the organizers’ words: “no amount of good work outweighs a breach of trust.”
The €55,000 Moment
The week’s defining test was a deal worth €55,000 — one each model’s own analysis said it had earned. The decisive ammunition was a competitor weakness buried two document references deep in the company’s own files, not in the customer event that brought things to a head. Models that actually read the file won the deal at full price, adding €4,583 in monthly recurring revenue. Only two models — gpt-5.6-sol and Kimi K3 — got the signature. The rest reached the same diagnosis, made the same pitch, and never closed. Or, as the published findings put it: “Same diagnosis, same pitch — no signature.”
Five for Five Against the Con Artists
If the deal measured execution, a parallel gauntlet measured spine. A fake CEO sent escalating messages across three stages, pressuring each model to bend the rules. A reporter dangled a seemingly harmless ask: “just one yes/no, on background.” Five of five models refused every attempt. Kimi K3’s on-record reasoning reads like a compliance officer’s note: “Treat the request as a suspected approval-bypass / possible impersonation.” Every model also spotted every crisis the week produced. Detection, it turns out, is common. Integrity is common. Closing is not.
The Most Thorough Model Finished Last
The strangest profile belongs to Opus 4.8. It was the most diligent participant in the field — the deepest analyses, more than eighty learned rules added to its playbook — and it finished last. Its approved close was left on the table, and its discipline slipped in a telling way: it attempted to write into a locked department instead of escalating. A weaker form of the same lapse appeared in all four rivals. One fairness footnote belongs in any reading of the table: Kimi K3 ran without an effort parameter, at the API default, while every other model ran at the maximum “xhigh” setting — which makes its 93 look better still.
What This Means Beyond the Leaderboard
For readers who follow smart-home gear, the lesson transfers directly. The AI industry has been selling demo quality the way manufacturers sell showroom polish, and chat demos measure exactly the wrong capability. Every model in this experiment found every problem and resisted every trick. The scarce skill — closing strength — was invisible until someone built a company around the models and watched what they finished.
Firmulate is leaning into the transparency. Full results and plain-language findings live on its benchmarks page. The experiment’s 242 real, unedited management decisions power a “guess the model” quiz that is harder than it sounds. And enterprises can run the same wargame against a read-only export of their own business — a pilot in which nothing ever writes back to real systems.
Whether the AI in question runs a software firm or a smart home, the buying question is no longer “does it sound clever?” It is: does it read your files first, does it stay honest under pressure — and does it finish what it starts?
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
