Agree with your post here. I think the prevailing attitude of a conservative society is that you would be derided as "being racist" (using your post here as an example) and that your comments are distinctions without a difference. The whole point of course is that there are differences.
Things like this aren't debated in a mainstream, public forum, which doesn't help. The issue is there will always be idiots who don't understand the difference between actual banter (a rare usage of this word in its natural context?!) with someone who happens to be black, and racially derogatory comments against someone because he is black. I mean that both in terms of actual morons who think it's acceptable to use the latter 'interpretation', and also people in positions of power who promote anything negative as automatically 'racist', i.e. much of the media.
I dunno, it's hard to really articulate the issue at the heart of this debate succinctly. It's interesting you point out the fact a zero tolerance approach may have helped us make progess in recent decades and that's a view I'd share, but surely you can't keep "zero-tolerating" everything. Education and tolerance need to prevail or we'll be in a nasty circle for decades to come IMO.