Unfortunately this is one of those things which comes down to the ever-regrettable it depends.
There are numerous variables that are important to any specific situation of sanding epoxy and they go beyond the basic question of high/low speed being better in general.
Normally I would have asked what sander you're using (not just the type, but the exact sander ...
I only did the work less than a month ago so I am surprised that its already wearing down.
That's likely why.
Varnishes take time to complete the hardening process that converts them into their final cured form. This isn't just after they have dried. Broadly, it takes about a month1.
What this means is that after varnishing something ideally one should wait ...