First Tri coming up and not sure what to wear?

Sounds stupid but is it really that important to have a tri jersey?  To me swimming in my tri shorts and putting on a dri fit shirt and shoes for the bike would work fine?  

I guess my question is what is the big deal with tri jerseys, singlets, etc.

Thanks

I understand the improtance of time and the advantage of wearing a wet suit when it's cold but what about other reasons.