In sociology, and some other social sciences, we take a close look at sex and gender and identify how those concepts are quite different even though society conflates them together.
Sex, being the more biological dimension of body type differences, and gender, the more social construction linked to identity and social roles, both define us as individuals and both are built into the very fabric of human societies.
In our society, we have long had the assumption that sex has only two categories, male and female, and upon that we build gender into two categories, men and women. Biology informed us that those two sex categories are the way it is. Since we put sex and gender together, our society then has traditionally recognized only those two genders.