Yes, the errors by pitchers can add up. I had a very good fielding team, except for the pitchers, and their errors accounted for 28% of the errors made:
http://365.strat-o-matic.com/team/1413661I had three starters and one reliever (who pitched a lot of innings) who were poor fielders.
Paxton 5-51
Despaigne 4-44
Garza 4-27
Betances 4-33
I picked Paxton, Despaigne and Garza because I thought they were bargains, not considering their defense, nor, in the case of Paxton and Despaigne, their lack of ability to hold runners on. Betances wasn't a bargain but was both a poor fielder and could not hold anyone on, even with Salvador Perez catching.