Swim issues

You aren't engaging your core properly.  You need to support your legs/hips with your chest if that makes any sense. 

You don't have to be so buoyant that you're like a natural life preserver to swim well. Even the densest, most muscled men with the biggest legs can swim well if they learn the correct body position and balance. 

You should do some float-type drills, where you lie flat out in the water, face down, arms stretched out forward, and minimally kick. Your goal isn't to go forward - your goal is to feel that flat position in the water. If you are getting tired from doing it, you're doing it wrong - it takes minimal energy. Turn your head barely to the side to get a breath - if you can do it with half your mouth in the water and one goggle still in the water, that's good.

After you get the floating mostly down (won't take too long), I'd actually recommend doing some pull buoy work with zero kicking. The reason for this is that you are almost certainly wasting energy with your kick - your kick is being used to keep your legs up, not to propel yourself forward, which is why you're getting so tired. Learn to do pulls with a pull buoy and then you can slowly re-add your kick - I think you'll be surprised how hard the coordination will be since you will almost certainly have a significant stroke error that requires you to kick significantly to offset it. 

Basically, you need to learn body position, and also simultaneously break your dependence on using your kicking to keep your legs up. 

Buoyancy helps, but not as much as you think. It's best that you just assume that you are fully buoyant enough given the right training, since that's the true reality of the situation. If it makes you feel any better, I'm a very lean male with 'thunderthighs', and am short, so I'm the epitome of someone who should 'sink like a rock' (if I exhale, I do!) But even with that limitation, I have no problems with the face-down float drills and swimming. 

If you ever want to see how much faster you can swim with added buoyancy, you can test it out with a pull buoy or wetsuit, both of which will add enough buoyancy to your lower half to not even kick for it to stay up. People often go faster, but it's typically no more than 5sec/100 if properly timed in a pool. It's not going to make you a 2x better swimmer, sorry.

And comparing yourself with others' curve of improvement is fraught with danger. Some people have a ton of talent, and some do not. I've seen the rare talented adult-onset swimmer literally go from 2:30/100 to 1:20/100 in open water racing in 6 months, whereas I can only hit 1:20/100 in a pool after years of hard adult-onset training. 

I have bad shoulder flexibility, if I try to keep my hands straight out I front of my head my legs will sink.  If I let my hands drop so that they are 8-12 inches under water, below my head, I float with much better balance.  I also find that I can start the catch & roll process a lot easier from this position as it helps me to keep the elbows high and I'll generate more power with the stroke.

Then the answer's pretty clear - you definitely have to work on body position without kicking! You should find swimming without a wetsuit much easier once you get this down. 

You can keep your legs up with either no kick, or a minimal kick that is close to zero. 

See my prior post about the float drill and pull buoy no-kick pull swims. Those should be helpful.