I actually went on a karate training camp when I was in my younger days and we were heading into a lake in our gi's. The locals were laughing there butt's off , they thought we escaped from the asylum, after we practiced our forms we started sparing and they stopped laughing.
Point is, don't stop doing something you like just because some people stare as if they haven't got tv's.