I don't know how brittle is delrin. I use a bit of HDPE for odd parts. The bad is that it is not possible to glue. The good is that HDPE is fairly elastic and very tough. So my first thought is to make an interference fit. I know Delrin/acetal is not as elastic, and also it's hard to see how small, that part actually is, so take this with a grain of salt:
E.g., take the insert and spin it in a drill while cutting a series of extremely shallow rings into it with a dremel cutting disc. Almost just distress the surface. Drill the hole with the next smaller bit. File/dremel the hole and chamfer the end of the insert until you can just tap/press the insert in without forcing it too badly.
If you get it just right, the bond will be plenty strong. But if you get it wrong you'll crack the part, of course. If it were HDPE, anyway, you would possibly have a very hard time to get the insert back out if you wanted to. Even as slippery as it is.
For this purpose, perhaps I'd grind 2-3 deep and wide longitudinal channels on the insert, too. To prevent it from spinning, of course.
You can possibly file out the hole (to clean it out) and fill it in with a piece of compatible plastic and a soldering iron, and try again.