I found a latent gap between its documented scope and its scoring pipeline.
Tool-call responses weren’t supported, but the response parsers could erase them:
The OpenAI adapter used message.get("content") or "". A tool-call response with null content would become "".
The Anthropic adapter kept only text blocks, dropping tool_use blocks.
The scorer excluded explicit errors, but accepted empty strings.
Given those samples, the scorer would see identical empty strings: one distinct output, byte-identical results, and mode share 1.0.
That would measure the stability of the fallback not the tool calls.
To be clear: this was traced in source, not reproduced in a live run. Current request builders never forwarded tools, so existing cases couldn’t reach this path. The maintainer checked all 563 recorded non-error samples: none were empty, and no published benchmark was affected.
The fix enforced the documented boundary: reject cases carrying tools, mark empty non-error completions unsupported, and exclude them from successful samples.
The broader lesson: preprocessing can erase the behavior you intended to measure. If “unsupported” becomes a valid-looking default, a reassuring score can hide the missing measurement.
How do you distinguish unsupported responses, parsing failures, and genuinely empty outputs in your eval pipelines?
[link] [comments]