 | I ran a small side-by-side coding experiment with GPT-5.6 Luna and DeepSeek V4 Flash. The task was to generate a playable browser-based camouflage game from the same prompt. Both runs were made through AIHubMix using the same API gateway. The recorded results were: - GPT-5.6 Luna: $0.10, 691 seconds, flawless first run - DeepSeek V4 Flash: $0.02, 729 seconds, successful on the third run The result looks simple: DeepSeek was 5x cheaper. But that conclusion depends entirely on how retries are accounted for. If $0.02 was charged per attempt, then the observed cost to reach a successful result was closer to $0.06. If the displayed figure was cumulative, then the 5x comparison is valid. This distinction matters because real AI systems rarely pay only for successful generations. They also pay for failed calls, retries, validation, human review, and downstream recovery. A more informative benchmark would report both:
It should also separate visual success from functional correctness, maintainability, security, and edge-case behavior. This was only one task, so it is not evidence that either model is generally better. A proper follow-up would use fixed model versions, identical settings, automated functional tests, at least 20 trials per model, and median and p95 reporting. What should be the primary metric for production AI systems: lowest inference price, first-pass success rate, or expected cost per verified result? [link] [comments] |
Predictive technology: computational social science