When I say a-gender I mean it in the sense of a-theist: I simply don't believe in gender. As I've posted before, I recognize that there are certain clothing styles, mannerisms, hobbies, etc. that are typically branded as "masculine" or "feminine", but I consider this labeling to be 100% arbitrary. It's handy that I happen to fall more on the neutral-to-masculine "side" for current American culture, but this has absolutely nothing to do with my body. As far as I'm concerned, one can be agender if they have DD breasts, are 4' 10", and wear makeup and tutus every day, and one can be agender if they have a flat chest, are built like a linebacker, and wear muscle shirts and ripped jeans every day. Gender is not sex.
Whenever I distinguish between gender and sex this way, I have to distance myself from the TERFs (trans-exclusionary radical feminists) who say that we should abolish gender, but then proceed to blatantly discriminate against trans women by claiming that they are actually males. TERFs see sex as fixed and unchangeable. I do not. Some primary and secondary sex characteristics are more difficult to change than others, but even if one does not seek (or cannot access) hormonal or surgical treatment at all, the brain determines one's true sex. I am male because that's what my brain says that I am. A transsexual female is female because that's what her brain says she is, whether she can afford the treatment to make her body look and/or function more like a typical female's or not.
And whether or not she or I look like a typical example of the sex that we have come to know that we are, we deserve equal and full access to the appropriately gendered facilities when no unisex options are available. It's no one's business but a doctor or sex partner's if a trans woman has (or once had) a penis or not, so that's no excuse for keeping her out of a women's restroom, locker room, or shelter. Really, it's no one's business what body parts anyone has under their clothes, or under their skin, unless they need access to those parts for performing some vital function - again, medical professionals and consensual sex partners only.
So when I say that I'm agender, I don't mean that I want a body that has no obvious sex characteristics; this is why I do not identify as neutrois. I am male, and those are the sex characteristics I desire, whether or not I'm realistically able to get them through medical means. Being addressed as if I were a female really bothers me because it invalidates my identity and exacerbates the dysphoria that comes from having a mismatch between body and brain. Being addressed as male is far preferable to me for that reason, given that very few people understand or are willing to use gender-neutral forms of address. I hope nonbinary gender is more widely accepted in the future, but I don't have the energy for that fight right now.