Swimming 1 mile 2x a week enough?

Enough to complete an Olympic distance triathlon - certainly.

Enough to race to your potential in a triathlon - probably not. That would call for more structured workouts and much attention to technique.

Great job losing so much weight. You probably lost a lot of muscle as well, so I would suggest you revisit your stance on the gym. There are lots of different gyms and equipment. But if you want to tighten up your upper body, your going to need to work out, at least pushups...

Also do you do intervals in the pool. Because our bodies respond to intensity not just frequency.

By "tighten", I assume you mean not look flabby? Swimming 1mile twice a week will not be sufficient. It's all about heavy, load-bearing workouts; so get used to weights, weights, and more weights. Do the gym, crossfit, pushups, pullups, abs/abs/abs everything you can. 

Keep the fat off and build up as much muscle as possible. Even then, if you're like me, you'll never get rid of all the loose skin, so there will be some flab in the gut, under the armpits, and under your chest. For me to get a six-pack, I have to hit the abs extremely hard and stay so lean that a vein pops out on my forehead. It's quite depressing but that's the price for letting yourself go in the first place.

Swimming 1500m/1650y twice a week will increase some upper body strength, but not very much and not enough that you will see any visual results. You would only be conditioning your muscles to allow you to make that distance. If you want to see significant improvement in upper body strength from swimming, swim an average of 4500 yards 4-5 times a week and you should be good to go (I used to swim 5x a week/year-round but I'm also just pulling these numbers out of the air, but you get the point).

You can gain upper body strength without joining a gym. At home you have your own floor for push-ups. Purchase a pull-up bar of some sort and install it in your home, like the archway of a door. Purchase hand weights - this opens up a variety of different upper body exercise you can do (arms, shoulders, etc.).

You just STARTED swimming and can swim a mile???  Wow, I wish I had your genes.  It took me a long time to get up to that distance.

Nice!

I saw some changes in my body shape when I began swimming for fitness and I was already fit, did some light strength training, yoga, pilates etc. I remember my friend (who also started swimming for fitness) and I discussing the differences we noticed: broader shoulders, lost a few pounds.

Also, another friend of mine who was strictly a runner trained for an IM last year. Her swim form was awful (not to be mean, she got some instruction and improved a lot before her IM). She swam with her head out of the water a lot at first and always used the same side of her body to lift and breath. Her shoulder on that side of her body was noticeably more built up than her weaker side.

IMO, if you are coming from no upper body workouts or previous swimming to doing 1 mile 2/week, I think you will notice some changes to your body.