Whenever I try to make a hole slightly larger with my cordless drill, the drill bit catches and wanders, and doesn't stay centered. Normally I use twist bits, but I'm pretty sure I would have the same problem with most (or all?) other types of bits (e.g., brad point, spade, hole saw, Forstner). I know I could fill and redrill the hole, but that would be a pain, I'd have to wait for the filler to dry or cure, and I'd have to re-mark the center.
How can I quickly and easily enlarge a drilled hole without moving the hole or having to resort to some other tool like a drill press (which often wouldn't be practical) or a router jig (as mentioned in Enlarging a stepped through hole -- which would not work for small holes)?