I had that some times when I started out serious spearing. There are several factors that you can work on:
- lack of comfort in the water. you know: salty, rough, currents, bad viz. this can only be cured by more water time, lots more, a good buddy helps too
- fear of the unknown. you know: water monsters, sharks, urchins. Again, need more time and a good buddy.
- dark thoughts brought about by the act of apnea itself: this is tougher, but happens to all divers. you dive, all feels great, you are down a while and suddenly these dark thoughts enter "ooops, i've been down too long. can i make it up. swim. panic. long way to go. dark feelings" etc. This I believe can be overcome in two ways: lots of hard pool apnea training, and, as previous two points, lots of ocean time, with a good friend/buddy!
Good luck