Why do non-pagans consider pagans weird? It would give an excuse for intolerance.
Sounds about right to me. Plus i think a lot of people are naturally inclined to fear things they dont understand, this combined with the commons misconceptions floating around (devil-worship etc) makes it easy for people to assume all pagans are crazies without digging any deeper...