It's not just dirt. Oil is so well filtered by modern filters that something like 95% of the dirt in suspension gets filtered even in the most common filters, like base line fram or store brand filters. The better filters only add 1 or 2 percent additional filtration and useually have better bypass valves, which don't filter anything. Essentially it would be possible to change your filter every so many thousand miles and leave the old oil in. EXCEPT... However, engine oil is made up of other things that get depleted and consumed in the process of running your engine, called the add pack.
That's the little guys that float around and makes nice with your metal parts. When these buffers, neutralizers, lubricators and fortifiers are depleted the oil can start to do damage to your extremely tight tolerance engine, think of metal parts slamming into one another at very high speeds. And the extra heat created by the additional slamming called friction. Oil does a LOT to cool your engine and transfer that excessive heat to the cooling system which is detrimental to your engine.
Also,there is a thing called acid. It has a corrosive effect on your engine. As your oil ages acids accumulate in the oil and eventually become determent to the engine and your happy little party turns in to a bad acid trip! (see what I did there bad acid trip... LSD, I kill me!!)