The problem is everyone discussing it are operating under different definitions of racism. They all agree racism = bad, but they're not agreeing what exactly that is.
Some people are arguing with the base definition that racism = discrimination on the basis of race. If that's the definition you're using, then yeah, saying black people can use the word but white people can't would clearly fit the definition of racism, and would therefore be bad, as per above.
Other people are arguing with the base definition that racism = material persecution of a race. Under that definition, it's much harder to claim that it's racist to say black people can use the word but white people can't, because it's a word that's been largely irrelevant to white people, so its exclusion from their vocabulary isn't material persecution.
Still other people subscribe to the definition that racism = the belief that one race is superior to another. Under this definition, actually saying the word would only be considered racist if saying it revealed an underlying belief of superiority.
All these people are arguing across each other, seeing each other as unreasonable, because clearly x is racist, because it clearly fits the definition. So it's bad.
