We paid a lot of attention to our choice of heating system when we built our house. We ended up choosing a Geothermal heat pump system. (which is effectively the same technology you'll find in a refrigerator, but then reversed) This effectively means three holes of 100 meters deep were drilled in our garden, in each of which four tubes were put. (vertical ground loops) The tubes were connected to each other prior of being put inside the ground, effectively making for a 400 meters long circuit inside the ground. Then the three circuits were connected to each other, resulting in a circuit of 1200 meters. After the tubes were put into the ground, a cement compound was injected into the drilled holes, so nothing would ever touch the tubes.

In-house, floor heating was put everywhere throughout the house. Then, the tubes were filled with water, with a bit of glycol added to it.
The system is as simple as it is brilliant. Water flows through the underground conducts, effectively reaching the temperature of the earth at that specific depth, which is always about 6 to 10 °C (40 - 50 °F). That water then gets compressed by the heat pump, which effectively raises the temperature of the water. One circuit is heated up to about 30 - 35 °C (85 - 95°F) and flows through the heating tubes under the floors. This is more than enough to heat up the whole house to a nice comfortable temperature of 20 - 24 °C (70 - 80°F), or more if necessary (like in the bathroom). The other circuit is heated up to 55 °C (130 °F) and is stored in a 180 litre boiler for hot tap water. I had my doubts in the beginning that 55 degrees would be enough, but it turned out to be MORE than adequate, to bathe, to do the dishes, anything you need it for really.

Of course, this system works best when the house is insulated VERY good, which is what we did: 10 cm (about 4 inches) of the most efficient polyurethane insulation, which works very good. (in total, our walls are about 40 cm thick - all brick).

This system has a high startup costs because of all the materials that are needed (LOTS of tubes!), and the drilling of the holes also doesn't come cheap. But after that, you get a system which is very efficient. The heating pump works electrical, so obviously my electrical bill goes up. BUT... I do not use gas, diesel, wood or any other of the classical combustible fuels. That's why our house also does not have a chimney. Absolutely NO risk of CO intoxication or fire hazard, since no fire is involved in producing this heat. (our fire/home insurance loves this, and this is reflected greatly in our annual bill!) My monthly electrical bill is about the same as what my neighbour pays... but mine includes my heating and hot tap water production: his does not. The system is also maintenance free for its entire life cycle. (which is about 25 years)
Another big benefit: in the summer, the system can be reversed and the floor heating can actually be used to cool the house, at a fraction of the cost of what air-conditioning would cost me. (since the water is cooled via the earth) This works VERY well and I would want to be without it any more. Last summer, when the outside temperature was about 30-35 (85-95) degrees for weeks, this system had no trouble of keeping our house at a steady 20 - 21 degrees (68 - 70 °F)... at only the cost of a circulation pump of a few watts that needed to run.

This system is essentially capable of keeping our house at a more or less constant temperature of 21 degrees all year long, at about half the price of what gas or diesel would have cost me. I never need to worry about setting the thermostat or turning the heat off when I leave the house: the system is always running because it would consume more energy when I would turn it on/off all the time then to simply let it run at a constant temperature (again, the refrigerator analogy). Since the heat is radiant floor heating, it's a very pleasant and comfortable heat as well.

You can see some pics here of the holes getting drilled and the setup. This is not our house, but these people worked with the same firm as we did, so the used method is identical (in fact, the workers on the photos are the same who did the drilling at our place .

We simply LOVE this system and are very happy we ended up choosing it. It also takes a lot of the stress off because once it's paid for, there's no need to fear the 'dreaded monthly heating bill' any more.