Nerds don't just happen to dress informally. They do it too consistently. Consciously or not, they dress informally as a prophylactic measure against stupidity.
-- Paul Graham
I don't have the brains to be a nerd, but I have the aspiration. When I won a position in the Writers Of The Future contest and there was this big event in United Nations, I was the only male in the whole hall who turned up without jacket and tie.
I just have an allergic reaction to formality of all kinds. It seems to me to all too easily become a cover for a lack of some kind. Formal speech can cover that you really don't like the other person, and formal dress can cover that you suspect that you really don't qualify for being there (whereever "there" is).
And for places, say restaurants, which won't let you in if you are not formally dressed, well... I guess they are trying to cover up that their exorbitant prices don't really pay for anything much.