Hurting sentiments

A while back I wrote a tweet about the Pope's condom comment. A friend advised me to be aware of "hurting sentiments". Unfortunately, I have never been able to reconcile freedom of speech and the hurting sentiments (HS) concept. Why is it that we need to worry about HS when hurtees (the ones whose sentiments were hurt) have collectively caused more harm to this world than some random bloke like me blurting out a random quote? I refuse to feel apologetic or to keep quiet for fear of HS when the other party has actually been responsible for physically hurting human beings (actively or passively).