Relief Pitchers, Avoid LHB and RHB Setting

I have two middle relievers. One is better against righties and the other is better against lefties. Overall, however, they are both better than my worst 50 cent "mopup" reliever in either direction. They are both set to "middle man" in settings. Because they are much better in one direction, I have also set them to avoid LHB and RHB respectively. I am concerned that the 2 middle relievers will be used for one or two batters and then HAL will bring in my Mopup pitcher during a close game because there are no other options in the middle innings. Is this a legitimate concern and should I remove the "avoid" settings on my middle relievers? Thank you in advance!