Besides the inherent dirtiness of most outerwear (how often do people wash hats, coats, gloves, scarves, etc vs. shirts and pants). It also gives off the non-verbals of the person being not comfortable or relaxed (as if they are about to leave).

The same thing happens with shoes. They are outside gear and a lot of people around the world (including me) don't allow folks to track shit all through my crib by wearing shoes indoors. My kid plays on the floor.

Same with sunglasses. A lot of people actually like to make eye contact with the person that they are talking to.