Remembrance Day (which also goes by the name Armistice Day) is actually on the 11th November, but poppies are worn in and around the second Sunday of November.
Specifically, the trend started to remember the end of World War I in 1918 (apparently it ended at 11am on the 11th day of the 11th month).
The poppy was chosen as a symbol of remembrance because many British soldiers fought and lost their lives in the fields of Northern France and Holland, where entire towns were obliterated, and yet fields of poppies continued to grow - bringing some color and joy to an otherwise miserable and horrifying situation.
Nowadays, the symbol of the poppy has come to represent support for the British armed forces in general, and remembrance for any troops that have lost their lives in action.
This has sparked some debate, as some people see a big difference between the act of going to war against Hitler's Germany and some of the more recent campaigns the British military has been involved in.