I generally have a little bit of cash on me. Ten or twenty dollars is pretty normal. I just like to be able to use cash if I stop off somewhere for food or beverages; it helps me to budget when I have the cash in my hand rather than using plastic.
It's never been an issue for me.
And really, I'm a lot more afraid of someone getting my plastic and personal information than a little cash. If someone takes a 20 off of me, that's all they're getting. But if they get my license, they know where I live. And if they get my plastic, they can run up big bills; I know you generally aren't responsible for what happens with stolen cards, but it's a hassle to deal with it.
Overall, I'm not really concerned about someone taking 10 or 20 dollars from me.