If I have a four man starting rotation with endurance ratings like SP9*/RP3, will it affect their stamina or effectiveness as starters if I also use them in relief?
yes, it will. however, i doubt HAL will allow you to use them in relief if they are your only SPers and you are using a 4 man rotation (not counting your 5th scrub pitcher). a team link would allow for a better evaluation of your team.
Using them in relief definitely will hurt unless plan on using Sutton as a fifth starter, presumably only at home. But if you do this on the road, whichever of the 4 main starters you use in relief will make his next start at a lower F rating, and your bullpen isn't good enough to try that.