I reran the benchmark. The deterministic result reproduced exactly — but the model-related metric tells a different story.

After the discussion on my previous benchmark, I reran the verification capability benchmark and inspected the results more carefully.

The benchmark contains 66 cases and measures 9 capability dimensions of a deterministic verification engine.

I ran two fixture-based benchmark executions:

publishable-v1-selfcheck publishable-v1-repeat2 

Both runs used the same commit:

f38cba58f4c0b108ae53cc2eb2a50ff9e4e806e8 

And both produced the exact same result:

66/66 cases passed.
0 failures.

Every measured capability dimension passed 66/66:

• Claim binding
• Evidence graph integrity
• Deterministic calculation
• Rule application
• Contradiction detection
• Missing evidence detection
• Verification outcome
• Reproducibility
• Auditability

So, within this benchmark configuration, the fixture-based deterministic result reproduced exactly across both runs.

But there is another number in the report that I think is important:

model_assertion_correctness: 12/24 

The benchmark explicitly treats that as a secondary reference metric, not part of the primary deterministic capability score.

And I think that distinction is becoming the central lesson of this work.

Deterministic verification capability ≠ Model generation correctness ≠ End-to-end production reliability 

A system can be internally reproducible and deterministic while still depending on a probabilistic model that produces unreliable, ambiguous, or incorrectly structured assertions.

That is why I'm moving away from treating the benchmark as one number.

Instead, I want to separate:

1. Can the deterministic verifier correctly process canonical benchmark inputs? 2. Can the model produce correct claims? 3. Can those claims survive parsing, normalization, canonicalization, and binding? 4. Can the complete production pipeline work end-to-end? 

These are different questions.

The rerun gave me more confidence in one thing:

The deterministic benchmark result is reproducible within the tested configuration.

But it also reinforced that reproducibility of the verifier should not be confused with correctness or reliability of the model that feeds it.

The next benchmark version will therefore focus on identifying the first invalid state for every failure:

Model output ↓ Parsing ↓ Schema validation ↓ Normalization ↓ Canonicalization ↓ Claim binding ↓ Evidence graph ↓ Deterministic verification ↓ Final outcome 

Rather than just:

PASS / FAIL 

I'm interested in whether others working on LLM + deterministic systems would benchmark these layers separately.

Would you treat deterministic capability, model correctness, interface/contract integrity, and end-to-end reliability as separate benchmark scores? Or is there a better framework for measuring this kind of architecture?