Somewhere in the last year the phrase world model stopped meaning a system that represents how things behave and started meaning any video generator with good marketing. What bothers me is not the word, it's that the evidence never changed to match it.
Look at how the last few launches were argued. Black Forest Labs put out FLUX 3 last week and the headline evidence was a preference test the lab ran on itself: its video preferred in 77% of comparisons against Runway Gen-4.5, 93% against Luma Ray 3.2. The fine print calls it a preliminary evaluation of an early candidate during midtraining. No methodology, no sample size, no rater pool, no prompt set. Meanwhile the same class of system gets described as having some idea what happens when you knock a glass off a table.
A preference test measures none of that. It measures whether a person picked clip A over clip B in five seconds, on samples the lab chose to show them. Cherry picking isn't even the interesting problem here. Taste comparisons can't be rerun, so nobody outside that building can check in October whether the model improved or the sampler got luckier. What is a 77% supposed to mean three months from now?
A public benchmark number can be attacked, and that is the entire point of publishing one. Somebody runs it with their own prompts, gets a different ordering, and now there is an argument with evidence on both sides of it. Nobody can rerun a preference win at all.
I'm not asking anyone to regulate a blog post. My problem is that a vendor run preference test has quietly become the evidence base for a claim about physical understanding, and those two things are not measuring the same object. When somebody eventually puts one of these behind a robot arm or a driving stack, that 77% will not have predicted a thing about how it behaves.
[link] [comments]