IMO, I am only adamant about a test drive in the two following scenarios:
1. I have no (or almost no) experience with the car in question, which is typically a "new car purchase" thing (but because of my obsession with rare BMW's, I always test drive, especially if trying to decide between two of the same from different sellers)
2. I have an inordinate amount of experience with the exact vehicle in question, which typically means that it's a car I've driven competitively, have rebuilt (BMW E46 chassis, for example, especially the M3; Z4M Coupes; E34 540i M-Sport 6MT's and the 03 E39 version plus the M5 of both generations; E30/E36 M3's...), in which case I have a significant "baseline" from which to draw comparisons. Most recently, this saved me from buying an otherwise pristine M3 with low mileage, which drove like a dream until you start asking VANOS to do its thing; while a moderately difficult (many here could do it very easily) "repair/upgrade" to replace the BMW seals/etc w the Beisan VANOS components (which along with the subframe reinforcements, is something ABSOLUTELY EVERYONE with an M52TU/M54/S54 equipped BMW should have done, with the oil pump nut important to those of us who truly "beat on" our cars, but if you are tracking one such car you either can spend the money to have the nut fixed AND for the ATI crank dampener or you can rebuild/replace your engine: it's not an "if" kind of scenario, you will blow the motor... I see it like I see helmets, seats, etc, in that they are expensive and a pain in the rear (sometimes very literally), but it is also the difference between walking away from a 120mph+ crash (*waves*) and not surviving one at 1/3 the speed...)
As usual, the point I am trying to make is somewhere in the disjointed text above