Many people, myself included, have a bit of an issue with doing street photography, because a few people will get upset due to the general paranoia these days. And when it is security guards or policemen, who have particularly much hard-suppressed fear, this will often convert to anger and a desire to demonstrate their authority. This has been trouble for some photographers, even when they were fully in their right.
In most countries, if it is in a public place, you can photograph it legally. That includes children technically, but that's a very, very sore spot these days, and I doubt it's worth the trouble, sadly.
I read a tip somewhere, probably on The Online Photographer, which is: when you get confronted, you smile kindly and say that you're just an amateur photographer enjoying his hobby. And then you point to your camera and say something like: "oh, you'll love this camera, it's 12 megapixels and yet it's small enough for a jacket pocket, and it even has a 5x zoom..." Blah-blah-blah, full geek mode. You don't give him a chance to get a word in edgeways.
While doing this, you take a step forward, not too fast, to show them the camera. And after half a minute, almost anybody will quickly get the idea that you are no threat, you are just a very, very boring camera geek who they will want to walk away from very soon now. Show's over.