Pardon me, but I lost an ear at mile 2...

I have various skull hats many by Headsweats plus a couple of headbands and choose depending on weather which one I wear. You will heed to open up the adjustments in back of your helmet and re-tighten it as your head will be bigger with hat on.

Breathable headband for chilly.  Breathable balaclava for cold.

Sub-freezing temps I wear a balaclava PLUS a headband. 

I hate frostbite & black ice so generally don't ride when it's that cold.  I can get a better (& less uncomfortable) cycling workout indoors on a trainer. 

Aerohelmet = toasty warm

Teens or below is trainer time.

I use a Buff for extreme cold weather riding on the bike or motorcycle.  Although, I really don't ride the bike much in the winter.  They are thin enough to slide comfortably under a tight fitting motorcycle helmet or cycling helment.  I have 2 different types, one is a Polar Bufff (half fleece) and half spandex and is for 30's and below the other is all spandex and works for that 30-50 range.  You can configure these things in a hundred different ways depending on what you need.  Also use mine for cold weather running. 

All real training at this time of year is on the back porch (which can get pretty steamy, even when it's in the 30s at the start of the ride. But for bike-commuting I use a couple of different beanies, with different thickness and amount of head coverage, depending on the temps. With helmet adjustment it's very easy, and that's the one part of me that does not get cold (hands, feet, a whole 'nother story).

I wear a Wal-Mart helmet when it starts getting cooler. The cheaper helmets aren't as airy.
I wear a Halo du-rag under the helmet.
Every now and again buy.com has a special on 180 degree ear muffs and I've picked up a couple of their 3 for $6 shipped. That's what I wear on my ears for biking and running. Easy enough to take off if you get too warm but you still have ears for the whole ride/run.

If you have an REI nearby, they have skull hats/beanies to wear under the helmet.  I have one and it works great!

x2
I got mine from a bike shop. They are pretty thin to fit under a helmet but, if you're working hard, that thin layer makes a world of difference. The beanies work under a running hat too, to cover your ears.

When it is really cold I wear one of these.

http://www.performancebike.com/bikes/Product_10052_10551_1086492_-1...

Works great for both cycling and running in real cold weather.

Fixed it for you crazy people