I would never dive without both! Here is why:
Time matters!
-----------------
You need the timer so that you can know how long you have been under. How long you should stay under depends on your training. Regardless of what that amount of time is, you need to know how long you have been under.
Depth matters!
-------------------
Unless you have something pulling you back up to the surface or some type of device that changes your weight or buoyancy you will find that you become less buoyant the deeper you go. This is because the water pressure is crushing the air spaces on your body, making them smaller ( like your lungs ). Because of this, the deeper you go, the harder it is to swim back up. With my current personal scenario ( I am a beginner with a few years of experience ) I start noticing ascension difficulty between 40ft and 50ft ( saltwater ). Bewteen 50ft and 60ft the difficulty seems about twice as much as the previous 10ft. Notice that this has nothing to do with time. The ascension difficulty is only related to depth.
Scenario:
-----------
Lets say that your normal safe turn around time is 90 seconds ( I told you that I was a beginner ).
Lets say that your normal depth is around 60ft ( still a beginner ).
Outcome 1:
--------------
You dive down to 60ft. When 90 seconds have passed, you begin your ascension.
You've trained for this scenario many times. Conditions are as you expect them to be and you make back to the surface safely.
Everyone is happy and you have a good day of diving.
Outcome 2:
--------------
You dive down to 60ft, but the dive has gone smoothly and only 60 seconds have passed. You still have 30 seconds of safe dive time left, so you decide to shoot down to 80ft to see what its like down there.
You reach 80ft depth and begin the ascension at 90 seconds ( your safe turn around time ).
During the ascension you notice that between 60ft and 80ft it's harder to ascend than you are used to. As a result you burn more energy and oxygen than your are used to. You are also further from the surface so it takes longer to reach it.
Because of the extra physical difficulty and longer ascension time, you run low on oxygen and suffer from shallow water black out at a depth of 12ft.
If you are diving with a buddy you will need immediate rescue.
If you are diving alone you are dead.
You ALWAYS need to know BOTH your depth and your dive time.
NEVER diver longer than you should!
NEVER dive deeper than you should!
Both are equally important if you want to stay alive.
That said, it doesn't take rocket science to get the job done. Sure, dive computers can be expensive, but they are worth it! I use a Suunto dive computer that cost about $400 because I'm also interested in SCUBA. You can also just use a buoy with a rope hanging below it with depth marks. If you are diving near a cliff, cave or reef, you can use certain familiar features to roughly measure depth. It doesn't have to be overly complicated or expensive, but it does have to get the job done.
However, since I don't currently dive deeper than 60ft I use a simple $10 water proof watch with a timer function. It's never let me down. I've been using it for years. If you're a better diver than me, you might need a real dive watch rated for 40 to 60 meters.