but using it in the same context as when it has racial overtones is
"Boy" is used as insult. To men of all color. Just because black people got to hear it bandied about a lot by white slave owners that thought of them as subservient individuals, as they were purchased and bartered like property, does not mean it's racist. It's just insulting. It should be left at that.
The problem is that this is not true. While calling a grown man a "boy" can be an insult regardless of races involved, it takes a more sinister and historically racist turn when used against a black man by a white man. As I said before, it's not even up for debate.
Sure some white people use it in a harmless manner, but as Stoney said in most cases they know exactly what they're saying. For instance, one of my friends is from Switzerland. If we're playing basketball sometimes he'll say something along the lines of "I'm too hot, you can touch me boiiii" like Flava Flav. I've known him for years, there's nothing racial about his intentions.
On the other hand I've been to Howell, Michigan a couple times and had the term leveled at me with a pretty blatant racial intention.