I'd either leave it alone or get some JB Weld or something.
I made my own super ghetto speaker adapters for my Camry. Broke out the old speaker from the plastic frame and used high-temperature black silicone to glue the new speakers into those frames, so that I could mount them. Bought the silicone at Ace Hardware, good for 400 degrees or something. It'll fill a hole nicely.