I grew up living on a dirt road. My parents drove Honda Civics, and later on my dad had Jeep Cherokee and Grand Cherokees. Dirt roads are hard on suspension, just how hard depends on how well the road is maintained. Our road was basically a mud pit in the spring months, and when the ruts froze it was rough on suspension. Tires also take a beating from rocks, we had a few flats over the years, and your vehicle will absolutely rust faster. The sand and dirt will collect in areas underneath, and also kick up onto the paint causing chips where rust will start.
With all that said, it was such a beautiful place to live and grow up.