+1 for spraying with oil.
If you "hack out the real loose stuff" you will be rewarded with some holes, which are mother natures way of giving you access to spray the oil.
Your trim (which you should be able to wire/cable-tie/stick back on somehow) will cover up some off these holes, and/or you can sometimes use tape, foil or even Denzler tape (hessian impregnated with petroleum jelly, used by plumbers to seal leaks, in the UK, at least).
Many people use "bondo" or fibreglass for cosmetic repair. I think that's usually a waste of money, time and effort, and will probably speed up the rot, but it MIGHT be necessary to pass a safety inspection if there are sharp edges exposed.
This above approach assumes you can cultivate indifference to cosmetics. If you can't, you'll need a new car soonish.