In the $150 to $200 range you can buy a decent new mini-PC.
OP advised he has an ESXI host, and is running RDP sessions to a VM on that host.
Used Wyse Clients are dime a dozen. They have little flash drives in them usually 64gb. You can load almost any OS on them, and there is probably a stripped down version of linux that you can load with a USB key.
I have imaged these things both ways, PXE or USB.
Are you doing VDI or just doing RDP sessions to a VM?