First, it makes no sense to tell people who believe human rights are being violated to mind their own business. (See also: “Don’t like abortion? Don’t have one.”)
Second, views on abortion show no correlation to gender. Even if we could forever silence men on the topic, the abortion debate would remain intractable.
Third, it’s insulting (and not at all feminist) to suggest that men are incapable of considering the moral, legal, and practical aspects of abortion. More generally, it’s insulting to suggest thoughtful people of any gender, race, sexuality, etc., are incapable of considering situations that don’t directly affect them. How many men fought for votes for women? How many white people spoke out against slavery? How many straight people fight for gay marriage today? Such people aren’t just mindless minions manipulated by the relevant interest group; they are conscientious humans capable of empathy.
Finally, and most importantly, abortion does affect men. While abortion’s serious physical effects are absent, reproductive decisions (abortion, adoption, childbirth, etc.) have significant financial, social, and psychological consequences for men. This is true whether the man is pro-life, pro-choice, or ambivalent.
This is not to say abortion affects men as much as women. It is to say abortion affects men enough that they have an interest in the abortion debate.