Ideas are not People

My brother recently told me that I need to be more tolerant of religion.  I told him he had it all wrong, and was making one of the great liberal mistakes:

People deserve the same set of basic rights within reason.  Ideas are up for grabs.  I don't have to respect ideas.  Ideas have no inalienable rights.