Hi spud, I was a student at the uni in swansea for nearly 5 years (most of which was spent surfing/fishing/diving the gower). Getting good viz down the gower is a nightmare and subject to swell and wind direction.
Looking at the tides and surf forecast your best bet is probably going to be Oxwich bay. There is going to be a small swell running which could make things difficult around the worm, which does hold fish on the sw side. However the tides massive and the winds offshore so be careful if you do go there.
anyway back to oxwich, its very sheltered, and as the wind is forecast to be light then swing northerly, it should be dead calm. The point at oxwich holds bass and some flats, plus because its a huge tide the wreck will be divable which is awsome, its bigger than you would expect, with lots to look at and a cheeky lobster if your lucky.
in short if i was going I would try and dive oxwich an hour before low on sat and sunday. Dont park in the carpark, its a ripoff, park on the hill, then after your dive dry off and go for a cuppa/publunch at the oxwich bay hotel which is on the beach more or less, nice food, big woodburning stove/fireplace to warm up in front of. Hope this was of use to you, good luck.