I think accepting that as your official wardrobe is a fine idea, and you do have the dresses for non teeshirt/sweats occasions.

(Every time I think of you, I think of those dresses!)