Fix and Forget: When Problem-Solving Undermines Safety