It doesn't help that there are a good number of cases of horrible child abuse (some of which are commited by pedophiles).

But you're absolutely right, human beings are not rational. We follow our emotions and our gut instinct more than the analytical parts of our brains.