Photography is one of my-far too many-hobbies but one I enjoy tremendously. In addition to film, my main camera is fairly high end(Nikon D850) and 75% of the time it's paired with a 24-70mm f/2.8 lens(and I have other f/2.8 zooms and a bunch of fast primes to use with it).
To be honest, I'm not impressed with inexpensive P&S offerings these days. The high end ones-specifically the Sony RX cameras-are incredible but they're a lot past that price point. The inexpensive end of the market has been shoved out by smartphones, which are pretty darn good these days.
$300 is not far off from the price point of a Nikon or Canon starter DSLR kit, especially from somewhere like Sams or Costco. Even though DSLRs are sort of on their way out, I'd consider one a MUCH better investment than a low end P&S. That also opens you up to whole world of lenses to expand the horizons/capabilities of your basic camera, and with the push away from DSLRs, the market if flooded with a lot of used equipment at bargain prices.
Size does matter, or more specifically sensor size. Even really good P&S cameras tend to have 1" sensors which are ~13mmx9mm. Low end P&S cameras tend to have smaller sensors. Post of the entry level DSLRs are ~16x24mm(APS-C format). Larger sensors tend to mean less noise("cleaner" looking images), better low light sensitivity, and more dynamic range. Of course there are some nuances to these, and often times moderate pixel count sensors outperform higher count ones all being equal, but that's still a good general rule.
That's just my take, though.