Not against their right as human beings or their sexual preferences - but biological men are not women and biological women are not men.
Well, biological men are not (born biologically) women, and biological women are not (born biologically) men. Obviously.
But we look at the terms male and female simply as an expression of "gender"- which is a separate concept from biologically-born "sex" of men and women - then the spectrum of human reality opens up somewhat.... and is revealed not to be as two-dimensionally black and white.
And none of this has to do with anyone's "sexuality" or sexual preference - which is a different element again.
In terms of public toileting - like in gendered sport - there are some complexities to sort out in moving beyond the strictly two-dimensional history that we have and which has informed the status quo.
Personally I'm an adherent of a three way split in things like toilets and gendered sports... along biologically born "sex" lines, but with a middle (non)"gender" ground - that could accommodate the intersex part of the biological spectrum, as well as accommodate (other people's concerns with) transgender individuals.