The whole reason Nazis are bad is because the violence they espouse may come to pass. Real violence is the evil we want to avoid. Denouncing Nazis is at its root a denouncement of violence. Real violence is worse than hypothetical violence, because without real violence, hypothetical violence is inert.
But real violence must be fought with real violence. So if you see a Nazi participating in real violence, punch away. But if you're the one injecting real violence into a situation that has no real violence, you're the bad guy in the situation. You've escalated a situation and now they have the right to defend themselves with real violence as well. You should never be the aggressor.
The only exceptions to this I can think of is immediate calls for real violence (e.g. "Let's go kill Bob at 4:00 at X address!"), where real violence has all but manifested itself; and continued harassment in someone's personal space where they try to get away and you won't let them so they push or punch you away, like in Buzz Aldrin's case.
