I understand it's a matter of taste vs effort
Actually not as much as people tend to believe.
Sanding to 320 grit may itself be too high, and above 400 you can definitely run into problems. Some woods burnish when rubbed with higher-grit papers and this can prevent the proper penetration of finish, which helps ensures a more even colour and the formation of a proper bond between it and the wood. Burnished wood can resist finish, even to the point that it will bead on the surface sometimes and to fix this you need to re-sand at a coarser grit to 'open' the surface of the wood again.
What determines the grit you need to sand to is not so much personal preference but rather whether you are finishing the wood and with what.
Oiled finishes require a more meticulous approach to final smoothing because they don't build a film and after oiling minor surface imperfections are magnified, not hidden.
With a film-building finish (shellac, varnish, lacquer) you get a filling-in of minor surface imperfections, including sanding scratches. This is why you can get away with sanding up to a coarser grit and still achieve a very smooth final surface.
Using a film finish, depending on the species, even 150 can be high enough. But the normal range to finish sand is between 180 and 240.
If you do not intend to apply a finish — which can be used for dense hardwoods, particularly tropical species with a naturally high resin content — then you can sand to very high grits indeed because you're actually polishing the wood itself. Even beyond the 600 you mention, I've seen the results of sanding past P1600 and you do get increasing shine, up to a limit dependent upon the density and resin content of the wood.