The difference is 2 homosexual men/women having physical relations is based on one concept - consenting adults.
Goats are not humansand therefore do not give consent.
How much perversion will you allow?
What if some people say that they are not attarcted to men or women, they are only attracted to goats. Will you allow that too?