It's kinda more efficient to keep the house at the same temperature all the time in order to avoid that situation.
Not so.
The only reason you put heat in a house at all is to replace the heat that has been lost through radiation, either into the ground or into the air surrounding the house.
The amount of heat lost through radiation is proportional to the fourth power of the difference in temperature (in degrees Kelvin) between the radiating body and the outside source that is absorbing the heat. Lessening this difference greatly reduces the amount of heat loss.
It is more efficient to minimize this difference whenever practical by allowing the house be cool (or in my Alaskan example, downright cold) when not in use.
tanstaafl.