This is exactly what I hear from people that I know, who aren't afraid to speek their mind on the subject.

I've worked around children for 20 years and I can't tell you how many times that I've heard "your just prejudice" as an excuse for whatever circumstance that arises. I agree, stop feeling guilty or let someone make you feel guilty for something that isn't real. Buy the same token, don't prejudge people, or use race as a reason to slant you viewpoint on a particular situation.