I always set Closer user to Maximize in Team Strategy.
Unfortunately, if you play many consecutive close games, he will eventually enter the game at "F0" or something close and usually blow the save. This scenario will likely repeat the next day, if another close game!
What setting can I tweak to prevent or at least minimize the number of times this might occur?
How should I set the "Do not remove before" option?
Definitely set a second closer. That alone should solve your issue.
Also, with my best pitchers, I set the F-rating for Don't Rel. B4 to F8 (some managers go a bit lower) and leave the setting blank for the vast majority of my pitchers.