It would depend on the circumstances and the individuals involved. For example, if you belonged in a white supremacist group, it would obviously work against you even if you claim that it wasn't a hate crime.
Now, I can understand that. A guy with a tattoo of a swastika on his forehead beating down on someone who isn't white... could probably be a hate crime. But what if he beats up on a white guy?
But the fight wasn't started by you calling him a taco. You didn't attack him simply because you hate gay people.
The people outside of the bar where the fight took place didn't know that, all they heard was me call him a taco. And that's the statement they give to the police.