Honestly,

both sides

have pretty solid arguments.
On the one hand, who you are definitely affects how a word is received, and communicating a message to be received is the whole point of language. If someone who's fat is calling another fat person "fatty," it can be taken as a term of endearment. Or in high school when my girl friends would call each other "slut." If a skinny person calls a fat person "fatty" or a man calls a woman "slut," it's by default seen as a slur. Who you are does matter.
Furthermore, it is very easy to censor yourself. I never cuss at all. I don't have a problem with people who do, but I can say from experience that not saying crude/harmful words is pretty easy.
On the other hand, most people have been raised on the notion that discrimination on the basis of race = racism. That definition is being altered by some people, but for a lot of people, they were taught from a very young age to
never discriminate on the basis of race. Making the n-word race exclusive is racist, by that definition that many were raised on. So it makes their instinctive "This is wrong!" alarm bells go off when they see discrimination on the basis of race.
And the argument can be made that making the word race exclusive is counterproductive. If the goal of progressive efforts is to make race a non-factor, it's pretty hard to see how making race a really important factor accomplishes that. It could be argued that it's like surgery, that you have to harm the body a bit in order to help the body in the long run, but a lot of people don't see it that way. Seeing how race relations have just gotten worse and worse over the past decade without any sort of progress toward a long-term post-racial world being made, it does seem that a lot of the well-intentioned progressive stances are counterproductive, harming the chances of reaching the long-term goal by being too shortsighted in their policies.
I don't really have a solid stance on the matter, because both have solid points. I do think that if you fall in the "equal for everybody" camp, the only tenable position is no one should say it, because it is undoubtedly a harmful word. The "everyone should be able to say it" camp doesn't have much to stand on, because it requires completely dismissing the harm the word does.