Zone 4 and 5 Training

What are the implications of training in zones 4 and 5?  What happens physiologically? I have read that training in zones 2 and 3 are essential to building a stronger base (aerobic range; exertion below lactate threshold correct?).  I have a hard time keeping an appropriate pace (bike and run) that doesn't feel unbearably slow in zones 2 and 3.  So, what happens when someone has higher than recommended times in the higher heartrate zones?  For instance on an hour ride this afternoon, to keep a reasonable speed (around 20mph), I stayed in the 4-5 range for around 50min (170+bmp; 36yrs).  This is pretty typical.  Am I working against myself?  I hear a lot about keeping in lower zones but don't know what happens to endurance and performance in higher zones.  Thanks for any direction...