Let me presume to offer 'expert' opinion. Having swum the English Channel both frontcrawl and backstroke. I see two issues here.
First, assuming you have never formally trained as a swimmer (as most freedivers) and can only swim recreationally in calm water for less than 20 minutes. Put this into perspective, you are a person who would not be capable of swimming ashore from say, the Cyprus barge in Howards comps but could swim ashore from Loics dive site in Ville Franche.
Maybe your standard of swimming is ok IF you keep your fins on. In which case, you might be able to swim 400 yards to get out of immediate danger, chase down a loose bouy, swim a buddy to shore. Remember this though, whenever you really need to rely on your swimming ability to get you or a buddy out of trouble, there will be a very high chance of secondary trauma that may make this emergency swim a much more serious event. ie: your dive buddy has been lost at the bottom of the rope, or floated to the surface with no lanyard and has been seen 200 yards away, face down. 200 yards means 2 minutes away for a seriously competitive swimmer or, 3 to 4 minutes away for an ok swimmer with fins. By then, you will approach your buddy totally unable to offer assistance until you have recovered first. What if you were doing pull downs with no fins, what difference would this make to your ability to perform a rescue? What if your dive line is half a mile off shore? Do you really expect to make it to shore, bearing in mind you will no doubt try to sprint the whole way, maybe 15 minutes for a trained and sprinting swimmer, then run (how far ?) 400 meters in a wet suit yelling for assistance? Let me say, there will be very few amongst us capable of reaching the shore in any fit state, if an emergency situation coincides with this need. Always assume you are injured, recent lmc/blackout , cramp, jelly sting (shark bite...and dont say that wont happen after my experience with an oceanic white tip 3 days ago, eye to eye at 12 inches, luckily his mouth stayed shut. Or Martins experience as the same 8 footer barged past him. Needless to say the videos only last a few seconds each and I dont know how to post them here). So the first issue is a question of the quality of safety you can offer as an untrained swimmer. Put yourself to the test and at the end of your next dive, swim ashore, fast. Dont take the boat.
The second issue assumes you are an accomplished swimmer. Not necessarily competitively trained or an active triathlete, but a fit, in shape freediver who trains in bifins and mono. You are a person who not only regularly trains in the local pool but regularly takes time out of freediving a reef or rope to swim or finswim a mile to the pier and back. Will this make you a better freediver......you bet. Put it this way, Freediving is like the Christmas turkey, but if you want to eat a banquet, you need to add other items to the menu. Our love of the water allows us to freedive, swim, scuba, and many other variations of expression including yoga and meditation, all of which compliment our main activity, we enrich our sport by becoming more proficient in a few of these other areas and they make our freediving more rounded. Put yourself a mile, or three, or six, off shore and swim ashore (have your dive boat keep escort if you need one quietly out of the way, you may get bored and tired and believe you may never reach the shore, you may retire early to your boat, but you will have caressed the ocean with a new touch, and she will reward you.
Just my tuppence worth. Now get Alun Georges opinion, he would have drowned if he fell into a swimming pool a few years ago.