I've had a pretty significant difference to the response I get from servers at restaurants between "t-shirt, pants, and sandals" and "collared casual shirt tucked into nice pants with belt and shoes", usually because they suspect I'll be able to afford tipping better. There's also a noticeable correlation between whether I'm near a college where where people are used to getting stiffed by poor college students and not.
But I've never been questioned at the grocery store for ID, and I'm almost always "extra casual" for that. Which is possibly fortunate, the photo in my ID is now 10 years old, and being relatively young I look pretty different. The guys at the airport always take a little time squinting before they make up their mind about me.