As with everything context is important. Years ago there was a push to remove the confederate flag from every state owned property in the state of FLorida. I know your first reaction is "Hell yes!", but nothing is ever that simple. The places it was used on state owned property also had the British and Spanish flags flown along side of it. They showed the history of the Florida and which country the land the now state of Florida belonged to. If it were the only one besides the US flag flying, of course it should have come down.
Our history isn't always pretty, but to completely remove any symbol of even the ugly parts, removes the reminder that we can and should do better. Yes there are people who use the symbol to perpetuate hate, but to take it from them isn't changing what is in their heart. And only makes it harder to spot the fools.