There are several things wrong with your assessment. First, far more heat is lost through conduction than radiation: the outside air touching your house, especially your windows. Second, you fail to take into account the energy required to reheat the walls, which have far more mass than the air in the house. If you're reheating the walls every afternoon when you get back from work, rather than minimizing the temperature difference between the walls and the (interior) air.
Let me re-state it more simply.
No matter if you lose heat through conduction or radiation or just leaving your windows open, your heating system must replace whatever heat you lose. The amount of heat you lose depends
entirely on the difference in temperature between your house (the heat source) and the outside (the heat sink). Think about it. If you were keeping your house at 70 degrees, and it was 70 degrees outside, you would lose no heat at all. It is temperature differential
and nothing else that determines heat loss.
You are correct that the walls have more thermal mass, but that is irrelevant. All that matters is how much heat is lost to the outside. The warmer the air inside, the more heat that will
radiate be conducted outside and have to be replaced. [Yes, you and Peter are quite correct about the conduction vs radiation issue. For whatever reason I didn't think about the outside air actually touching the house.
)
It is astonishing (and rather appalling) to put this question to Google and see the amount of misinformation out there being propagated as fact. Even a slight knowledge of thermodynamics (all that I have) proves that any opportunity to lower your in-house temperature saves heating cost.
tanstaafl.