Ok, let’s try extreme analogies.
Let’s say I am hanging around actively arranging plans to murder my neighbors for fun. In the middle of this, I decide I would like to have some sex. I find an amenable person and am talking to them as we walk to my home. However, during this conversation it happens to come up that they think murdering people for fun is bad. This makes me not want to have sex with them anymore, so I don’t.
I get to do that!
The thing I am doing wrong in this situation is ‘arranging plans to murder my neighbors’. This is bad. I shouldn’t do that. I can in fact legitimately be arrested for it. (As far as I remember law things). It does not mean I should experience rape, because literally nothing of the sort means that.
I can be doing something bad, which I shouldn’t do, and decide not to have sex with someone for a reason entirely stemming from that fact, and I still get to not have sex with them, because I always get to not have sex with people for any or no reason.
The bad thing I am doing is bad, and I still get to not sleep with people when I don’t want to.
Sometimes pain, and very rarely unexamined 
I feel this so much, and your example made me chuckle a little bit. tfw your one night stand is dtf (down to fuck) but not dtm (down to murder)
anyway
The way I phrase this in my head is “I can say no to sex for any reason or no reason.” That way, I don’t even have to burden myself with the mental effort of articulating a reason why. There doesn’t have to be a reason, because my decision is valid whether it’s based in inchoate instinct or capricious whim or careful, deliberate thought.
Compare this to some other decision I could make–people will get upset and baffled if I capriciously decide to, say, push my car into a lake. The difference may be that saying no to sex is never objectively stupid, because this isn’t a question that can be evaluated objectively. Deciding to have sex is so intensely personal that the decision has weight regardless of whether our whys and wherefores have been digested into language.
Like they say: we all have our reasons.
“I can say no to sex for any reason or no reason.” Yeah, definitely that!
Hm, I think at least part of the difference may be a) absolute personal rights, and b) to some extent if not a super relevant one to the sex thing in question, action vs lack thereof.
I have an absolute personal right to my sexual boundaries, so it doesn’t matter what my reasons or lack thereof are, I just get to not do sex things. (Similarly, I get to say ‘not hug people’ or ‘not let people telepathically into my mind’ (were that an existent thing) for any or no reason.)
And trying to make people have a reason (which, like you said also implies ‘something that can be digested into language’), let alone an ‘approved’ reason is a form of force toward doing-sex-things-they-don’t-want – it will cause more of that to happen. Which isn’t ok.
(And to some extent, though I think not super relevant to the sex thing, actions vs lack thereof – as in, it would be bizarre if people were like ‘but why didn’t you push your car into a lake?’. For a lot of things, ‘not doing thing’ is just a default, it’s the thing that happens when other things don’t. Which, like, shouldn’t be used to stand in the way of people’s rights to do things they have rights to!
But we are not beings with the whole world in our minds at once – there’s at least some extent to which even as the answer to ‘why did you do thing’ can be ‘I felt like it’ or ‘autopilot’, the answer to ‘why didn’t you do thing’ can just be like ‘that didn’t even come to mind in any way’ or ‘my attention, which is not infinite, was entirely elsewhere and that didn’t enter into it at all’.)