Unfortunately some people are just narrow minded like that.
I have a theory that treating ginger people differently has its roots in our innate biological impulse to stick to "our own tribe" and show hostility or weariness of those who don't look like us.
It's the same impulse that protected us all from being eaten alive when our ancient ancestors were living in caves.
Obviously most of us have evolved since those days, and are able to use rational thinking to override our more primitive instincts. Some of us still struggle with them though, and that leads to the kind of behaviour you described.