I think the fact that Ray makes a point of reading out, very clearly, the e-mail addresses of those who don't agree with him is indicative of the guy's attitude. I never heard him read out the e-mail addresses of those who agreed with him when I used to listen. He wields his show as a weapon.
That's one of the reasons why he wouldn't get too many e-mails/calls from people who disagree with him. He can't accept different opinions, and resorts to attacking the person.
The other reason is that anybody who disagrees with him or dislikes him is unlikely to be listening to his show in the first place. I know I don't anymore.