Matt's answer covers things well but I wanted to add something about increasing efficiency since you say you are often faced with the task.
Preparing in advance
I think you should approach the job from the outset with the goal being to reduce the amount of sanding to a minimum, irrespective of whether it's by hand or using a powered sander.
So for example if you were making multiple small blocks from a longer square-section board the long sides should be fully smoothed before any final sawing to size. This means they can, in theory at least, be planed or scraped surfaces, reducing the total production of sanding dust to a minimum which is a worthwhile goal.
This will result in only the new sawn faces requiring sanding, two per block rather than all six. Going further, if you shoot the end of the board before cutting off the block you reduce the number of surfaces requiring sanding from two to just one :-)
Tips for sanding by hand
When we read "sanding by hand" there's a natural tendency to think of holding the workpiece in one hand and a sanding block in the other or perhaps even just a folded piece of sandpaper. But you'll have a much better sanding experience if you keep the sandpaper still and move only the workpiece, which leads on to the idea of jigging up your sanding. This will give greatly improved efficiency and accuracy.
So the first 'jig' I think is worth making is the sanding board. This can be nothing more sophisticated than a full sheet of sandpaper taped to a flat piece of MDF, particleboard/chipboard or plywood (as long as the surface of the ply is very smooth), or if budget allows using self-adhesive paper on a smooth substrate such as glass or melamine-faced board. You can also build a dedicated board for this, with means to hold the paper securely on two or four edges:
[Source: sanding hook from this Instructable]
[Source: this blog entry from user rance on Lumberjocks]
Note: while you can move the workpiece over the sanding board any way it's often beneficial to use a circular motion rather than sanding back and forth, this can help to prevent rocking the work and introducing an unwanted curvature to the face you're sanding.
Next is the sanding fence, which is just a low fence attached square to a board that has been faced with sandpaper. Pressing the workpiece flat against the base while moving back and forth along the fence it's very easy to keep your edges square by something that's not at all easy when sanding by other means, particularly with thin stock.
Another way of approaching the same thing is the sanding shooting board. This is exactly what it sounds like, a shooting board where you use a sanding block/sled/plane to run past the edge of the workpiece rather than a normal plane.
The above picture shows a commercial sanding sled in use, but you can of course easily build a sled or sanding plane from scrap wood.
Although you can do this on the edge of your regular shooting board it may be best if you make a board specifically for this as even after carefully brushing clean some abrasive particles are left behind and will inevitably scratch your plane the next time you go to shoot normally. The scratches will only be superficial so whether you feel the need to make two separate boards is up to you.
Improving the cuts to minimise the need to sand
The saw matters hugely in how smooth your fresh-sawn faces are. I don't know which saw or saws you're commonly doing the final cutting to size with but a fine-toothed razor saw (e.g. 32tpi or higher) or a somewhat equivalent pull saw (more teeth per inch, but different tooth geometry leaves a cleaner sawn face) can each leave a surface so smooth that you might in some circumstances decide it doesn't need to be sanded at all.
In addition to the type of saw, if you gauge or mark your cut lines around the wood with a sharp knife prior to sawing all four sides of the cut can be left much cleaner, reducing or removing the need to sand any edges too.