Air flow "in" is also restricted by the exhaust. Fuel injection era vehicles & OEM panel filters far exceed CFM requirements for a given displacementor and will do a better job of keeping out the dirt.
Brakes get hot, that's how they stop the car. Nothing unusual going on here. The drum will heat, expand, and cause little, if any, excess wear to the lining.