perhaps the presence of seals means there are fish around, or maybe the presence of a seal means the fish have swimmed away???
We have lots ( too many?) seals in some of the areas I dive, particularly within range of the Farne Islands where up to 4000 Grey Seals live.
This may be different in other areas but you can tell if a seal or seals are around as the fish become very skittish and even good Pollock often stay under the cover of kelp rather than swim in the open.
If its just an individual that moves on it still takes a good while for the fish to settle down. If its a few of them hanging around you'll really struggle to find very much.
I've never had a problem with them apart from the odd bit of posturing/ showing their teeth but they don't worry me too much and they can be fun to swim with but not if you're after fish.
It never fills me with much confidence when I see them, in fact I would rather go elsewhere if I see them before I get in because they really decrease any chance.

