But that's where you're wrong, it does.
If we want to get "super factual" on the matter, let's discuss an entire village of male-only species. People are going to have instinct and needs. Everyone will be homosexual. People will not say "no, this is wrong, I need a female" because they were not raised near or have any sexual feeling to anything other than their surrounding "experiences" growing up.
That's still an environmental factor, and although it seems very unlikely for the above to ever occur, I'm sure testing the theory will give you said results.