Rape is not about the physical violence. Objectively a rape victim might end up with far less "physical damage" than someone that got beaten up... In some cases the actual physical damage is minimal (and no.. I'm not minimizing the seriousness of rape I happen to see rape as unfavorably as murder in many cases). I see it as the main issue being akin psychological torture. Not only the feeling of hopelessness and lack of control during the act itself, but in almost all cases the effects far outlast the initial insult.
Society has a special place for sexual acts. Do I think there is an objective reason why this should be the case? Not really.
I think for millenia, humans have used rape to "damage the goods" (the good beings, wives and daughters of those fallen in conquest) and obviously considering women as "goods" is barbaric, but that does not mean that the stigma of being raped is not real or that the act itself is not about control.
I do realize that consensual sex in which 1 person sneaks a hole in the condom might not carry the same visceral reaction in many of us but it is a very similar act to violent rape.