It IS a black/brown/person of color and white issue. You always have to consider the power dynamics and historical context. I'll put it a different way... as a woman I can call my best girl friend a bitch and mean it as a term of endearment - but if you, as a man, do it... prepare to get your face slapped.
I once went out to a Salsa dance club with my white friend Tiffany who proceeded to drink too much and thought it was hilarious to walk through the bar yelling out "Juan" or "Jose" to see how many people turned around to look. I was like, WTF are you doing you idiot?!? I told her she was going to end up getting a well deserved beat down if she didn't shut up and I got her out of there immediately. Proper names such as Tyrone/Jerome/Pocahontas can be and are used as racial slurs.
@dviss1 has probably called some African American friends the N word over the years but I don't ever say that word because I'm not a person of color and it has a completely different meaning when it comes out of my mouth.