Serious question for people working with ML systems that act autonomously. We often optimize for correctness, confidence, or expected reward. Yet many real incidents come from systems behaving exactly as designed, while still causing irreversible damag…