Thanks all for the help. I finally figured out what the issue was. Initially I was thinking along the lines of KrisZ, that the "ears" were hanging up in the bracket. So I disassembled the brake today, and cleaned up again the ears, reinstalled all the parts. Sure enough the caliper and pads were sliding back and forth easier than ever, smoother than any brake job I have ever done. But when I bolted up the caliper to the pins, and pumped up the brakes, all of sudden the caliper wasn't releasing fully. So I knew it had to be related to the pins. I had burped the pins so they were sliding all the way to the bottom, but I noticed one seemed to go in further than the other. Turns out, the upper pin and lower pin are different lengths, and I got them mixed up on the left side only. I reinstalled the pins in the proper location and now the brake finally releases with almost zero drag. Road tested the car and everything seems back to normal.
Oh, and I still had my old pads kicking around. They actually measured between 2.5mm - 3.5mm. So I could have run them a bit longer, but it was low enough for me. I found the pedal was starting to get a little low on the car and the transition from regen to hydraulic brakes was more noticeable. The new brakes made a higher pedal.
So if anyone does a brake job on a Gen III Prius, the pins are not identical, and don't get them mixed up.