For small cylinders as described (4mm dia, 15mm long) the premise in your OP will work, but the drill press used needs to be accurate.
My recommended execution of your idea:
Assumes negligible play in drill press and use of good brad-point drill bits. Drill press table should be as close to chuck as will still allow the exchange of bits.
Cut a piece of 18mm MDF about 100mm wide and 100mm shorter than your drill press table's diameter. Mark centre. Clamp down with centre mark at drill bit centre. Drill a 4mm hole to 2mm deep, and without unclamping drill your desired bore diameter right through the 18mm MDF.
Unclamp, set aside and retain. This is the top half of your jig.
Clamp a second piece of 18mm MDF to the drill press table (100mm longer than the first, which should equal your drill press table diameter). Clamp at very outermost available places.
Bore a 4mm hole to 11mm depth, then drill a 2mm hole clear through the MDF.
This second MDF chunk is the jig's lower half. It remains clamped in place until operations are complete.
Blow out any dust from drilling, and place first cylinder in place.
Place jig top half onto top of work, clamping in place using the extreme ends of the jig half.
The two jig halves will compress the work from top and bottom, preventing rotation while drilling.
Drill desired bore into work.
Unclamp jig top half, remove work by poking up from below through the 2mmm hole.
Blow out jig.
Repeat.
Since this is a very small object, go easy when drilling and don't try to hog it out all in one go. You'll get a good number of cycles out of the jig if you take care to remove and re-clamp gently.