How do I avoid or remove the scuffing?
Basically you can't. Until you get to really high grits (way above P1200) you will see the scratches and these are not suitable for de-nibbing.
How do I smooth polyurethane after it dries but still maintain its finish?
In general you don't just sand, you then polish the surface. Wet-sanding will reduce the scratch pattern somewhat as well, compared to sanding at the same grit dry, but the main 'secret' is that you then use some polishing agents to buff/polish the surface.
Traditionally the polishing of a dry finish was done with something like rottenstone. This is still in use today by some, particularly traditionalists, some high-end woodworkers and professional finishers/restorers.
You can in theory do this entirely with abrasive paper or film, but you have to go to extremely high grits and these can be quite costly. While that is how some do it it's really not necessary, today I think most people use something like an automotive finish polish e.g. Simichrome, Flitz or T-Cut to rub out the finish to its final high gloss.
How do I actually sand a nib out of polyurethane?
You can use the abrasives you're already using, but another tip you might like to try is to use the rough side of brown paper. This is just abrasive enough to take most nibs off without the risk of rubbing through the finish. It will leave the surface visibly less glossy, although if you buff hard you can get some polish back (essentially giving you something like a satin or semi-gloss finish with care).
The ideal situation might be considered to be where you don't need to sand any nibs or imperfections, and if you set up a dedicated finish room or zone where fluff or dust are kept to a minimum that's theoretically possible. But in practice this is something you will have to deal with sometimes so it's good to have a process ready to tackle it when it does occur.
Note: wiping on varnish can nearly eliminate problems with dust nibs (also brush marks, runs and drips). The downside is that because you're applying the varnish so much more thinly build is much slower and it can take many more coats (7+) to get to a fully-built finish.
Also does the technique vary for gloss vs matte/satin?
You're not supposed to work over the surface of reduced-sheen varnishes, anything from matt to satin or semi-gloss should ideally be applied and left alone.
At the pro or high-end level the goal with these is to get a smooth build using gloss and then apply one coat, two at the outside, of the reduced-sheen varnish as a final coat. Obviously you take great pains to try to ensure you don't have any dust in the environment before the final coat goes on.
A better way to achieve a sheen below fully glossy is to scuff the surface of gloss varnish. You can easily get anything from a subtle satin to full matt depending on the coarseness of the abrasive used (powdered pumice, fine grades of steel or bronze wool, Scotch-Brite or an equivalent).