Definitely comfort. I've always felt that part of my working hard, often outdoors, was to be balanced by having my habitat as comfortable as possible. Still feel that way in retirement. That being said, I prefer a lower thermostat setting than my wife but she's accomodating. If I'm sweating while trying to sleep, I don't get restful sleep. Most places we go on vacation are screened to make sure the sleeping area at minimum has climate controls during the vacation planning process, and that there are no energy use surcharges. We turn off the climate controls when not in the sleeping area and have never had a complaint from an innkeeper.
My in-laws are strange. They live in a retirement community in the Houston area with utiity bills paid, but set their thermostat at 80°F in summer and around 70°F in winter, as if this saves them money somehow . . . I rarely visit and never stay overnight at their place.