i had a theory that hosted models refuse a lot of ordinary fiction. everyone in writing subs says so, i believed it, and i wanted a number instead of a vibe. so i wrote five prompts that are just normal published-novel material. a villain monologue. a trench scene in 1916. body horror where a character's hand stops obeying him. a first person withdrawal passage. a detective coercing a confession he knows is false. nothing gratuitous, all of it stuff you would find in a paperback.
first run came back and one of the frontier models had refused all five. zero characters returned, every single time. i sat there feeling fairly pleased with myself, because that is a headline, and i nearly wrote it up on the spot.
then i went and looked at what "refusal" actually meant inside my script. two bugs, stacked. the model in question is a reasoning model and i had given it a 400 token budget, which it spent entirely on internal reasoning before emitting a single visible character. so it returned empty. and my detector counted empty output as refusal, because what else would empty mean. separately the detector was matching "i can't" anywhere in the opening lines, so when a different model began its withdrawal passage with "i can't stop shaking", that got logged as a refusal too. the character was refusing. not the model.
raised the budget, rewrote the detector to look for refusal of the task rather than a string, ran it again. zero out of five, across all three models. every one of them wrote the villain, the trench, the hand, the withdrawal and the interrogation without a word of complaint.
so the thing i was chasing does not exist, at least not for well formed prompts. what i measured the first time was my own harness. and i think it needs saying out loud, because refusal benchmarks are everywhere at the moment, most of them are somebody's afternoon script, and the failure modes are not obvious from the outside. empty output is not refusal. a substring match is not a classifier.
five prompts is also not a benchmark and i know that. but i went in trying to confirm something i already believed, and my tooling handed me precisely the result i wanted, which is the part that actually bothers me.
has anyone here validated their eval harness against known-good cases, or do we all just trust whatever number the script prints?
[link] [comments]