Not sure exactly what you're asking...? What do you mean by "physical limit"? Losing consciousness from hypoxia? (At which point autonomous respiration would usually begin - though that's not 100% guaranteed under certain rare circumstances, so...)
The vast majority of those without much training will end breathhold when it gets 'too uncomfortable' (usually due to high CO2) - there are some studies on apnea breaking point, for example
see here. Some who have trained breathhold a lot are able to continue until blackout (though would only do so very occasionally).
So are you asking about the factors that influence the time it takes to reach hypoxia/blackout? If so, I'm curious why you want to know...?
Even then, the answer is pretty similar to before: given two people with reasonable health, and no current major contra-indicators, it's very hard to tell who will black out first, and could even vary hour to hour, and day to day, depending on the state of the body at the time.
In the end it's all to do with keeping a supply of oxygen to the brain - and that depends on the initial amount of O2 in lungs and bloodstream and rest of body at start of breathhold, how quickly that O2 gets used up by the body during the breathhold (activity, heart rate, constriction of blood flow to various parts of the body, relaxation, metabolism, maybe even fighting off some mild unnoticed infection, etc.), the diffusion of O2 from the lungs into the blood (affected by partial pressure of O2 in lungs, which comes from current ambient pressure and percentage O2 remaining in lungs), diffusion of O2 from blood into the body (affected by amount of CO2 through the Bohr effect), and any constriction of blood-flow to the brain. There's also the extra oxygenated blood cells stored in the spleen and released by spleen contraction - and probably numerous other things I've forgotten just thinking off the top of my head.
I suspect the only way you would have any chance of a half-decent prediction between any two people with decent health would be to take lots of medical measurements just before the breathholds (lung capacity, blood gases, heart-rate, maybe recent nutrition, metabolism, spleen size, etc. -perhaps even doing numerous tests for signs of that mild unnoticed infection I mentioned! And even then you don't know how well their dive reflex will kick in to decrease O2 usage throughout the body, and how effectively the spleen will release extra O2 into the blood...)
Does that help at all? (And I'm still curious why you're asking...?)