Why is it.. (this question just eats at me)?

Because it is the ignorance of religion that is really annoying. The most ignorant of these religions are the ones who are the most vocal about converting people. They have no idea that religion has ZERO to do with God. Buddhists and Taoists have a philosophy that is live and let live.