Not all Americans were behind the War of 1812. The British army was many times larger, better equipped and experienced, as well as battle-hardened.
Although the British were also busy fighting Napoleon's powerful French army, many Americans were concerned that defeating the British would not be possible, and that war could be very costly and damaging.
There was also division on American states about whether the local militia's that operated at state level had to answer to the president or not.
There were certain states who felt their civilian militia should exist separate of the national armed forces, and when other states were engaged in battle, they would some state militias did not intervene.