Who's Cheating in AI Tests?

War of the benchmarks

Just hours after the release of xAI's Grok-3, a heated battle for the title of the best LLM erupted on the social media platform X, marked by accusations of fraud and denial of each other's achievements. This highlights a growing issue: traditional benchmarks are increasingly inadequate for evaluating models.

Shortly after, Noam Brown, the mastermind behind OpenAI's computational model o1, offered a solution. Instead of constantly creating new benchmarks for various metrics, he suggested evaluating new models based on their cost-effectiveness relative to performance. This is a smart approach, as cost is often an overlooked but crucial factor.

Lower costs allow for wider distribution, boosting the company’s brand while encouraging more people, including those with little or no prior exposure, to try out AI models. This is exactly why GPT-5 will be permanently available in the free tier—a brilliant strategic move by Sam Altman.

In the end, success isn’t just about having the best model but about achieving the widest distribution through effective PR. Despite fierce competition, OpenAI stands out with Sam Altman’s exceptional marketing skills. It’s no coincidence that he was President of Y Combinator before becoming CEO of OpenAI.

Today’s Sponsor

The future of presentations, powered by AI

Gamma is a modern alternative to slides, powered by AI. Create beautiful and engaging presentations in minutes. Try it free today.

Flying Car Production Set for 2026

Alef Aeronautics successfully tests its Model A flying car that drives and takes off vertically. The company targets production by early 2026.