You might actually be able to fix this using white or yellow glue but it would be better to use a gap-filling adhesive. The simplest option for this is epoxy. Virtually any epoxy will work here, so buying the cheapest 5-minute epoxy you can find usually isn't a false economy (even dollar-store/poundshop epoxies are generally fine).
You can use the epoxy as-is, or thicken it slightly if you want by mixing in a little filler. Wood dust is the usual filler so use that if you have any, but wheat flour will work in a pinch.
Although you might be able to push them home using a pencil or screwdriver I think you may need to use the cam bolt to correctly position the nylon inserts, but you don't want the bolt itself to get epoxied to the side of the hole. To prevent this it would be advisable to lubricate the shaft with a thin smear of wax, oil or grease. Any cooking or salad oil will be fine, you could also use mineral oil or baby oil if you have either in the house. Don't get it on the outside of the plastic insert.
You'll want to clean the hole out as well as you can before adding in the epoxy so that it isn't sticking to the previous glue you used but to the wood itself. If you have a drill bit that's a tight fit in the hole run that up and down the hole a few times with it spinning. If you only have a bit that's slightly larger than the hole that's fine to use instead, you'll re-drill the hole at a larger size but since you'll be filling the gap anyway this is no problem.
There's no downside to trying this first if you want. There's a chance that packing the hole out with thin slivers of wood (split matchsticks, toothpicks or bamboo skewers are all options) and then forcing the plastic insert in will give them enough purchase.
The projecting bits of wood can be snipped off with scissors.