Take a 99-04 Tracker/Vitara for a test drive. You might like it or, hate it, but atleast around here they are around half as much as a CRV of similar age/condition.
Ours has done pretty well, one front axle seal, a starter, and a O2 sensor have been the repairs so far. Its got a timing chain, transfer case, ladder frame, rwd with solid rear axle, with the same mileage as a CRV.
Its a bit smaller inside than a CRV or Liberty.
I find in our forestry road adventures, 98% of it could be done with a small fwd car, but there's always a washout or steep rocky hill that does need some half intelligent driving and 4wd and low range to go slow to make sure I don't break our ride home.
IMHO for a fun mild off roader you might as well get real 4wd and a transfer case, and probably a Tracker is the cheapest way to do that. The CRV is a bit better commuter, but the fact you have to spin up the front tires to engage the rears is not ideal even for easy trails. I know for what off roading I've done, spinning the fronts could have had me stuck at best or broken down and stuck at worst...
The RAV4's of that era have a 4wd system to lock in the rears atleast but research which years had a bad manual trans if your looking for that.