[Discussion] Fine-tuning vs. inheriting base model behavior — a case study with an abliterated Qwen base

Sharing this because the eval writeup raised a question I haven't seen discussed much: when you LoRA fine-tune on top of an already-modified base model (in this case, one with its refusal mechanism removed via ablation), how much of the resulting behavior is actually yours versus inherited?

Context: a LoRA trained for a narrow task (recalling internal codebase conventions) on top of an abliterated open-weight base, r=16, ~80M trainable params. Disclosure: this is from a side project of mine (ClawHunt), sharing the technical findings rather than promoting the product.

The interesting part for discussion: the adapter didn't add or remove any refusal behavior that's entirely inherited from the base model choice, and it's the kind of thing that's easy to overlook if you're only evaluating the fine-tuned adapter's task performance and not auditing what safety-relevant behavior came bundled in from the base.

On the eval itself, reported honestly rather than rounded up: passed internal benchmark gates but the improvement over the previous version wasn't statistically significant (McNemar p=0.5) — a tie, not a win. Compared against Claude Sonnet 5 and Opus 5 on the same prompt set, the adapter won on domain-specific recall (expected, since Claude never saw that data) but the two Claude models diverged from each other on hallucination-related failures in ways I didn't anticipate.

Curious whether others building on abliterated or otherwise-modified open bases are auditing for this kind of inherited behavior, or whether it mostly gets treated as a solved problem once you pick a "known" base model.