Do you teach high school or jr. high? What do you mean by "get physical"? We're not allowed to physically discipline kids in any way, as far as I know. 
Well, I don't mean punching them in the face or bending 'em over and whacking 'em with a ruler, but you're telling me you haven't seen the teachers give the kids a whack on the head with a textbook when they're asleep or that kind of thing? You can pretty much get away with whatever you want to in this country...not that I recommend it.

My first year here at the midyear conference, there was even a workshop on disciplining the kids and all the fucked up shit that the two ALTs giving the presentation have done to their high school students. My favorite story was when the guy told a kid to stop checking his phone repeatedly, then finally took it from him, put it in a plastic bag, and
threw the bag out the window (it landed safely on some bushes). The kid got up to go get it, and he told him "If you leave this class, you
BETTER NOT EVER COME BACK." The kid sat down, shut up, and never did a single bad thing again.
I've had to get physical before though. A month back, kid had drawn a swastika (not a manji, which is the symbol that goes in the other direction) on his arm and was showing me. I was like "dude, I'm Jewish. I don't want to see that...do you even understand what that means?" So then he gets up in my face with this other kid, waving his hands directly in my face. I told them both to stop very nicely. One kid backed off...all I did to him was tell his homeroom teacher, and he was made to apologize. The other kid, however, didn't stop. I even lightly pushed him away, but he came back and laughed with "that doesn't hurt! You're weak hahahah!!" I had kept my calm this entire time, but when he started flinging insults and talking to me like I was gutter trash, I snapped. I grabbed his arms and shoved him against the wall hard. Shit hanging on the wall fell down all over the hallway. I was fuming and said (in Japanese) "Huh? What? What the hell did you just say to me?? HUH?" Then I heard him meekly say "it hurts" and I let him go, as he dropped down to the floor. I stormed off to the teachers' room, found an English teacher and explained what happened, who then told his homeroom teacher.
The kid must have known what was coming, because he bolted from school. I saw him later on outside, sitting down in front of a building next to the school with a big smirk on his face, talking to some other kids. I stopped my car and rolled down the window, and he fucking bolted again. I turned around and went back to tell his teacher where he was, and when I drove back, he was sitting there AGAIN with a smirk on his face. As soon as he saw me, he took off running and hid behind some trucks. I think he must have thought I was going to kill him or something. After that, he didn't come back to school. The teacher called his parents, and they told me he was going to come back to apologize, but he never came to school. He also lied to his parents and told them that I started all this by pushing him around for no reason earlier (which no one believed)
A week later, I happened to be walking alone near the entrance to the school, and I see the kid ride by the school on his bike in plain clothes, like he's on vacation. He sees me and freaks out, but rather than run away, he comes up to me and meekly tries to apologize. I tell that's fine, now let's go hear that in the teachers' room. He tells me he doesn't have a uniform, so he'll come tomorrow. And of course he doesn't. I see him the next two days outside the school on his bike like before. Finally he comes to school with his father. He apologizes and the father apologizes too. I tell him finem whatever, etc. just come back to class already. He does, but it turns out that this kid doesn't do shit anyway...in ANY class. His parents don't think he needs to learn anything. He also went and broke a bunch of windows last year. I don't get why he's even still in school, but he hasn't messed around since then, and usually says hello to me in a weak voice if I'm walking by.
I didn't even mean to push the kid, but fuck, he got me legitimately pissed. That's never happened before...I rarely ever get angry, because these are just kids. They got their hormones changing and are getting over puberty and all that, so I know how it can be for junior high kids, but this kid just went over the line. There are other kids like him, more at my other school too. One kid tried to close a window on my face last year. I got super pissed but told his teacher, who fucking let him have it and that was that. That kid has attemped to fuck with me a bit this year, but fails miserably every time. He doesn't even attempt to do anything in class either. It's a shame, but fortunately this type of student is in the vast minority.