There are several issues here. The most obvious of which is the publicity accorded to the "hate merchants".
The quiet, peace-loving, hard working majority are too quiet and their voices are drowned out by noise made by the radicals.
The same radicals, are currently making things very difficult for an ex-colleague in Bristol.
She has twice received letters (anonymous) warning her that she is "living in a Muslim area" and asking her to "dress appropriately" (I cannot believe the lady concerned is dressing in any way immodestly by normal native UK standards.)
She has lived in the same house for 27 years and didn't "See apartheid coming in my own country." (to quote her own words).
Lastly, I think there is the problem of "Dislike of the unlike". Adherents to Islam are, generally-speaking, different in outlook, colour and attitude to the original inhabitants of this island (whoever they are).
If it is any consolation, the same intolerance has affected minorities and second and third generations of immigrants throughout history, and we usually get it right in the end. (Huguenots, The Flemish, The Jews, the Poles, the Irish have all found it difficult - and it clearly had nothing to do with religion for them.)
As you will remember from my contributions in the past, I am ambivalent towards all organised religions.