By focusing on your limiters during the off-season you can accrue your highest rate of return for training time invested and set yourself up well for next season.
Your training schedule should be based completely on the hours you have available to train, not necessarily on the number of hours you "need" to train.
My sister warned me about being used as a launch ramp by faster swimmers but much to my surprise I was the one swimming over those in front of me. I went around one woman, then another.