And I think when someone begs you, you should follow your instinct. Spontaneously, if you think you should give him, just give him the money. It shouldn't be rigid policy of not to give at all. Just analyze the situation, think, feel and act.
When I was in Karachi, on a lot of occasions when in doubt I used to give money. This is not a favor that you are doing to that person, you would just expect the return from ALLAH for that and who knows your intention better than ALLAH?