Emile Durkheim said that we only understand normality by working out what's abnormal, if it's not abnormal then it must be okay* - and that gives us a bit of latitude. We know that some human societies permit polygamy. We also know that some societies (a very few) permit polyandry. If we examine those societies then we see there may be good reasons for those rules. For someone in a western society they are bizarre behaviours but in those societies they are the norm - they serve a purpose. I'd extend that a bit and ask if a certain behaviour does harm. If it does no harm then what's the problem? I will allow that the inquiry in to harm must be extensive though and not shallow.
I read somewhere where someone wanted to marry a dolphin. I think I just reached a Durkheimian limit at that point. I don't think it's possible for a human to "marry" an animal. I have my limits

But if we all agree that it's far too extreme to permit marriage between human and animal, then can't we agree that marriage between two humans is okay? And if a group of humans wish to marry isn't that okay? And if they wish to bring up children in a group marriage isn't that okay? I mean, will it do harm?
*Durkheim preferred to use the terms "deviant/non-deviant".
Touche, well said. It was a difficult position and I don't know the answer either. The "What harm is in it?" is a good start I think. I don't know about the deviant/non-deviant definition though, we're back to who get's define it.
I go along the lines that every case is an individual one in which each individual must ask him/herself "What is this for? Why am I doing this? If someone is not capable of asking this question society needs to step in. If someone can honeslty answer I mean to do harm then society needs to step in? It's not perfect either.
So we agree it's not up to "unwashed masses" (joking) to decide. :-6