I had a 95 Camry with the same engine I bought used with 33k, with similar symptoms of blue puff of smoke on start up that started about two months or so later. I got it from a good dealer along with an extended warranty.
That photo illustrates how badly those engines could sludge so quickly. It wouldn't surprise me to learn the previous owner actually did reasonable oil change intervals, unawares of the festering problem. Toyota denied it was a problem until some years later when they gave some sort of extended warranty coverage, which I didn't qualify for due to my higher mileage.
(In 1998 there was no BITOG nor was I aware of the critical importance of the PCV on these very sludge-prone engines).
When the dealer pulled the valve cover, it looked like that, but still in the brown gelatinous stage. I had looked down the oil fill hole before purchase to see if any sludge, but Toyota has that baffle with the black crinkle finish there, so you can't see anything meaningful. Understandably, the warranty company wouldn't cover it.
Long story short, the dealer remanned the heads (a nice job, too), but I am not sure they cleaned the oil pump. You may want to try some other steps before doing the valve stem seals, but your thinking along those lines is spot on.
I drove that Camry symptom free for another 90,000 miles when the oil light came on and it locked up. I ended up replacing that engine with a used one, drove that one for another 50,000 and sold the car to my GF at the time. Just to be sure, I pulled the valve cover on that used engine (I wasn't going to sell her a mess) and it was spotless. I'd been using synthetic for both engines, probably Mobil 1 at the time.
There is another lengthy thread here in BITOG of someone working with the same engine.
I would clean what you can manually first, and pull the oil pan to clean the oil pump screen and anything else you can get to. My engine later had a rear main seal leak, which went right on the exhaust pipe = major stench at stop signs and red lights. The warranty covered that, but I have to suspect it was related to the sludge.
To pull the oil pan I think you have to slightly raise the engine to clear the front cross member (this may not be true on the Avalon, but it and the Camry share designs). If you have to pull the oil pan, the rear main seal would be really logical.
I'd also Kreen the bejeebers out of it (at proper dosing), changing oil filters at no more than 500 mile intervals, 2-3 times over.
After the heads were redone, mine didn't burn oil...maybe a quart every 4-5k, which I don't consider oil consumption at all. I haven't owned the car for nearly 10 years now so am a bit rusty on that point, but it wasn't much.
It will be a project, but if you're up to it could be rewarding. If you end up going with a used engine, pull the valve cover on it before you buy, to make sure. Free of this issue, these Toyota engines last almost indefinitely. Given the very low miles on your Avalon, that could be the way to go vs. the longer process described here. It depends on how much time and attention you want to give it as a project.
Keep us posted, and we loooove pix here