Have you read The Left Hand of Darkness by Ursula K. Le Guin? It hits a lot of the points you talk about here: the one human / 'stranger in a strange land' is a black man named Genly Ai, and he is envoy to a planet of aliens who are genderless/sexless most of the time. They choose a sex once per month as part of a fertility cycle; one cycle they could get pregnant and the next impregnate someone else. The story is a thought experience about how a society without a concept of gender would develop. One thing I found interesting is that the King's offspring are divided based on those born from his body vs born from a partner's body, making the society vaguely matrilineal (because children borne by the King are in line for the crown, because their royal lineage is guaranteed).