Polyurethane is typically my go-to finish, but I'd always just kind of done it how dad showed me rather than reading the instructions, sanding with fine grain between coats, or not at all if it's still "tacky." It always seems to turn out fine, but recently I read the back of the can and it said to wait 4 hours between coats (i.e. not tacky) and to use fine (000 if I recall correctly) steel wool rather than sandpaper between coats. Is this one of those "recommended but not required" sort of things, or have I been doing something which will bite me later?
If you are happy with the results, then continue as you have! Using steel wool or sand paper allows you to create a uniform surface for the next coat of polyurethane, which generally looks nicer, more professionally, etc.
In my experience, assuming the finish is dry, there isn't a difference other than the coarseness of the sand paper or steel wool. For some projects, a first pass with a coarse grade and then a finer grade gives a smoother texture to the last coat, especially if the last coat will be thin and glossy. This is most critical where the surface will reflect light, like a table top, since the reflections will show any imperfections.
And I have used both sand paper and steel wool on the same project depending what is on hand, generally sand paper first and then steel wool.
I have noticed some finishes (rarely) have a slight issue with the trace amounts of oil in the steel wool. I haven't seen this problem with polyurethane, only with very thin shellac. I haven't had issues with rust myself but glw's observation makes sense.
Either is fine. I tend to use steel wool more often and that is mainly due to the fact that I refinish furniture on a regular basis... and there are curves and grooves that sandpaper just can't handle.
That said if you are working on something old that has damage to it... sand paper can be a lifesaver because it doesn't snag. I have had to do significant rework because the wool snagged on a corner with a little gouge and it pulled off some wood...
So for curves definitely wool. For a big flat surface, paper works great too.
One other note, the higher the sheen the finer I go. For example with a gloss finish (which I would almost never recommend... unless you have a basketball court maybe) I would use 0000 steel wool whereas with a satin finish I tend towards 000.
My understanding has always been, that the reason for doing any sanding between coats had to do with the grains standing up. So after the coat dries, especially polly, the grain stands up and feels almost like sandpaper itself.
Sanding this smooth before the next coat will make the finish be smoother. I tend to use very fine sandpaper when doing the in between coat sandings. I will know when it's enough by feel, I'll run my hands over the piece checking for 'rough' spots.
The biggest problem you can have is if your sandpaper is too rough. It can leave marks that will show through the finish. The other thing not to do (which my brother discovered to his embarrassment) is don't use mechanical sanders, they can eat right through your coat of finish, letting you start all over again.
Sanding between coats of poly is not a must-do. It's not required for a proper bond to form (the most persistent myth) and it's not required for a good, flat varnish job.
Poly bonds perfectly well to itself, gloss surface or no gloss surface.
If you do want to sand to flatten and smooth off — for example if you're going for a super-glossy 'boardroom table' level of finish — then it's better to sand in one go right at the end (after a suitable curing period, which could be weeks or longer).
The reasons you may need to sand between coats are to "de nib", that is to take off dust specs and knock the tops off any bubbles in the finish. You can also sand to take care of drips etc., but proper application technique should avoid those. This is an area where 'prevention is better than cure' definitely applies.