My drill press doesn't have a "stop".
How do I make a drill hole the same depth every time?
A lot of people drill a hole through a suitable length of wood dowel (or small square cross section) and use that as a stop-collar on the drill.
You can also just use an external chunk of wood
One benefit of both these is you don't need an Allen key and can very quickly swap back and forth between two or more different depths (useful for e.g. dowelling)
Unlike some pieces of tape or rubber/plastic grommets, they can't be pushed out of position by overenthusiastic pressure.
Tape is what I would have suggested and continue to use but if you wanted something a little more robust my suggestion would be depth stop collars /nuts
Image from AliExpress
Easily removed off the bits and adjustable as well which tape would not be as much.