Never pull out your wallet for a beggar, unless you're in a crowed area I guess. I used to run into a lot of them back when I rode the bus daily. If I had change in my pocket to spare, I might hand it over, often not though.
It's not that I'm afraid they'll use money for drugs - I can care less; my tax dollars fund far worse shit than one homeless guy getting stoned. I just assume beggars will rob if given the chance, so I don't give them that chance. It's especially bad at a bus transit stop, jesus. I'd always make sure to just carry my ID in my wallet on those days, leaving everything else at home.