To give or not to give?

I had an experience two days ago while I was busking at the farmer's market, and I'd like your opinion on what you think is the correct course of action.

I was busking at the downtown market with my flute and an open case of money at my feet. I had been playing for about 45mins and so a fair amount of money had accumulated. As I was taking a break, a homeless man approached me and asked if I could spare some change. He seemed sober (and I couldn't smell alcohol on his breath) and was not "stoned" as far as I could tell, so I pulled $2 out of my case and gave it to him. I don't usually give money to people who just ask for money, but my money was sitting right there, so I did. 

Then he said "Please sir, can you give me at least $5, enough for a meal?". I said "Sorry", but he kept saying "please". I kept saying "Sorry", but he just kept repeating "Please, I haven't eaten in days". He wasn't being obnoxious, but he was just being very persistent, and finally I fished $3 out of my case and he took it, thanked me and off he went.

I'm never sure whether to give money or not. My kind of default answer is "Sorry, no" but I'm not sure whether that's the best (and most moral) answer. I've seen posters around the city saying not to give money to panhandlers, but to instead give money to homeless shelters and places that provide safe food and shelter. That way you know that it's going toward "healthy" stuff rather than drugs, alcohol etc. I think that's probably the best thing to do. But how could I in good conscience not give him anything when I had a full case of money sitting right there? 

So comment with your thoughts!

• Do you give panhandlers money? 
• Do you think we should just give money freely and believe that it's not our responsibility to know what they do with that money?  
• Should I have caved in and given him that $3?

Thanks for the thoughts.

Busking at the Farmer's Market

Every Saturday, my city has a Farmer's Market. It's a really neat place to roam around, with all sorts of great food and interesting creations. I've been going for as long as I can remember. Another cool thing about it is the buskers. All different kinds of musicians playing different styles of music on different instruments. I remember when we'd go the the market, I would always stand and watch the buskers, fascinated. Then my mom would give me a loonie or a toonie and I would happily go and place it in the instrument case. It made my Saturday morning.
About a year ago, I started busking at the market, playing flute. It's really fun (and makes good money!). But I honestly don't just do it for the money. It makes me really happy to just stand there and play music for people. And growing up with it, I can appreciate how neat it is to have buskers at the market, and know how much it adds to the ambiance. I love seeing little kid's expressions, as they stand and stare at me in fascination. And then their parents will give them money to put in my case. Although sometimes they need a little prompting. This morning, a little kid walked confidently up to me and was about to lighten my case by a few coins, until his mom walked over and said "no, thats his money", and promptly gave him some money to put in. It pretty much made my day.
I'll likely post more busking stories in the future, as they come!