I don't own a pair of sweat pants, and I only own one pair of shorts which I rarely wear. I guess it comes from work, and not wanting to kneel down on bare knees in the filth others have left behind, or having to yank dried paint out of my leg hairs. I generally just go with a button shirt, darker jeans, and a nice pair of shoes or boots in public situations like flights.
I do own flannel pajama bottoms, those and a t-shirt are what I generally wear around the house if i am not leaving for a while.
I am not opposed to dressing down, and have a nice (ish) suit I wear on occasion, I just don't ever have much call to wear it unless someone's getting married or buried.