My hope is that in general hand tools will not produce fine enough dust to be concerned with.
While it is now fairly widely known that the finer dusts are the most hazardous (as produced primarily by certain power tools) this can tend to obscure the fact that all wood dust is a potential health hazard.
This position is considered overly cautious by the standards of some woodworkers, and it is erring on the side of caution, but the one thing you want to be careful not to do is fall into the "it won't happen to me" mentality as we all know where that can lead*.
It is true that many people have gotten away with a working lifetime of occupational exposure to wood dusts with no apparent harm, but equally there is the occasional unlucky person who is exposed only to small amounts of dust periodically in their hobby woodworking and that is enough for them to develop an allergic response (which can quickly become bad enough that they have to stop using one or more species of wood completely, the reaction can be that severe).
Needless to say there's no way of knowing in advance where on the spectrum you might fall, so it does make sense to take precautions as though the dust is harmful to you. Because it might be.
So IMO any extensive sanding, even if exclusively by hand, should either be done with an efficient dust-collection system in place (e.g. over a downdraft sanding table) or wearing a well-fitting dust mask. Planing or chiselling don't generate any airborne dust to speak of so the risk there is too small to reasonably worry about. Hand sawing, scraping as well as file or rasp work you'll have to weigh for yourself based on the species you commonly use and how much of the work you're doing.
Why and when should I wear a dust mask?
What kind of wood dust is toxic/dangerous and requires usage of dust mask?
*Things like removing the safety guards and riving knives from table saws and never putting them back, building router tables without a guard despite it taking almost no time to fit one, running wood over an unguarded jointer holding the workpiece in the hands.