I think you made the right call on those occasions when you had your son with you. Unfortunately confrontation is confrontation, regardless of how petty the offense that starts it off. In fact there might almost be an inverse relation between the pettiness of the offense and the reaction of the perpetrator when confronted. Most people are conflict-averse these days.
I was in a coffee shop in Manhattan a couple of years ago, when a disheveled guy came in, stood at the back of a long line (directly behind my colleague and I) and started shouting profanities. There were kids, all sorts of people in that line, and everyone ignored him. This went on for about twenty seconds, when I, for no other reason than that it was morning, I was tired, and I didn't want to hear it, turned around and told him to shut up. He did, looking like he'd never heard the words before. Even more priceless were the amazed looks I got from the rest of the customers. I suppose he might have had a gun or a knife, but if we expect the cops to do everything then we surrender.