I'd say look at the body fat on most Olympic swimmers and you'll see clear evidence that people with very low body fat can indeed swim quite well!
Swimming well is a matter of form. I've seen plenty of people who *shouldn't* swim fast but do because they swam competitively as a kid and learned good form. Conversely, I've seen plenty of people who you'd think could rock the swim but don't because they never learned good technique.
I've done a number of triathlons as well as a few open water swim events. My 50th percentile triathlon pace is about 75th percentile in the open water swims.