It's funny how the only contrast is AC on or off. Vehicles built in the last 40+ years have variable speed fans, many with recirculation modes and ability to set temperature, and cycle on/off as needed.
How much heat your AC is removing has a lot to do with the load it becomes. If you are contrasting full blast and the AC was in good condition, windows down probably wasn't an option as it'd be too hot.
Regardless, it's not just dust and pollen that get in with windows down. There's rain (when you forget to roll them up parked) but most of all exhaust from everyone else, and in certain areas of some cities it also makes you a magnet for harassment (or at least more interaction than you wanted) whether it be punks or homeless people or hookers, etc.