You might just need to practice. Not actually timed myself holding whilst swimming, but I do 33m (one length of the pool I use). Also look through the threads here about doing exercise whilst holding your breath, I believe this can build up your tolerance to the carbon dioxide if you do it regularly. Occasionally I just hold my breath when I'm walking around the house. But obviously don't try doing apnea cycling, there's a few obvious dangers there. Another one is apnea weightlifting, but be careful there as there is the possibility of a blackout, and you could end up dropping the weights onto your feet or something.
As I said before, just look through the threads here, and you might be able to find some useful ideas, and keep practicising. Your times could increase as you practice.
Good luck,
Matt