I kind of suppose the reason is that a hand-held router has to be light-ish, so can't have a big 1HP motor, and so the inertia and smaller chip size (therefore lower cutting force) produced by a high-rev cutter allows a small fractional horsepower motor to be used.
FYI there are 1-2hp professional routers as well as production routers with motors in excess of 3hp. All are still hand-held, although admittedly the heavy and very powerful ones are beasts and need strength to use.
But it also occurs to me that perhaps wood cuts best at high cutter surface-velocity (surface feet per minute, SFM) and the generally small diameter bits used in a router just require the revs to get the required cutter surface velocity.
I don't think this is as much a factor as one might at first think. Wood can be cut extremely well at very low speeds if the cutting edge is very sharp (think paring with a chisel).
Router bits tend not to have edges that sharp, and the high speed does help compensate for this — although that's not the reason for the speed, the same speeds were around back when bits were commonly sharper because their cutting edges were made of steel.
One of the things the high rotational speed is yielding that's less obvious is a flatter cut surface. So for a given feed rate the interval between one cutting edge and the next coming into contact with the wood is reduced, meaning the surface is less likely to show a repeating cut/not cut pattern1.
But as basic as the following seems speedy cutters simply yield faster results. Not long ago a wide range of cutter heads were sold for mounting in power drills, for routing and other milling jobs (including panel raising if you can believe that). While they were apparently capable of good results by comparison they worked much more slowly and the advent of lower-cost routers that most home woodworkers could afford killed them off pretty comprehensively, which is a shame as I think they would still have something to offer2.
1 You get this is you use a feed rate that's too high, the bit's edges then aren't keeping up with the advance of the wood so slight gaps open up in the cutting action, yielding to scalloping. In some woods it can lead to terrible tearing too.
2 A couple of advantages for the home woodworker are that they were much easier for the user to sharpen (carbon steel and HSS, no carbide) and they would tend to produce more parings/shavings and less dust, certainly much less fine dust as routers do, which is a boon for anyone who doesn't have the means or space for a proper dust-extraction system.