When that dries, its gritty ... even after the 3-4 coat.
This is likely because the first coat of finish caused some roughness to emerge1, referred to as 'raised grain', and you didn't tackle it then. When this happens you need to sand after the first coat of finish has dried enough that it's easily sandable (about a day, sometimes less and sometimes more) to remove any of this initial roughness.
You can submerge any roughness of this kind if you just apply enough finish, sometimes referred to as "burying the raised grain", but this may require perhaps 6-8 coats of oil-based varnish applied at full strength with a brush so it's quite a commitment in terms of amount of finish and time needed with oil-based varnishes.
I'm looking for a REALLY smooth and glossy finish.
You could probably do with what's variously called 'perfecting the finish', see this previous Q&A, Leveling a finish/finishing the finish. Also see How do I achieve a "piano black" high gloss finish on wood? for further related info. Do note that generally this process requires the finish to be fully cured, so a wait of at minimum two weeks to as much as a month after the last coat of varnish is applied (some wait longer than a month, just to be on the safe side).
Depending on just how smooth you mean, you may also need to use grain fillers or pore fillers (often the same type of thing) for a completely smooth finish on some species. Anything with coarser or more open grain can really benefit from filling, the oaks are maybe the best examples but also ash and chestnut, and to a lesser extent the various substitutes for mahogany and black walnut. You don't need to use grain fillers on any softwoods (no pores), or on tight-grained/close-grained species like maple, beech, cherry or poplar.
Note: if you are grain filling you generally do this after the wood is stained, with a filler colour matched to the stained and finished colour of the wood, or a tad darker.
I thought one wasn't supposed to sand after a poly coat. Do I need to sand in between poly coats?
Sanding after the second coat of finish is a slightly contentious issue as many finish manufacturers recommend it, or state it's necessary, and numerous woodworking gurus do it and suggest it's needed, but of course you don't sand anything unless you need to sand it, and the simple truth is sometimes you don't need to sand between coats of finish..... in fact you can substitute often for sometimes here although how you apply the finish and how clean the drying conditions are (i.e. little or no airborne dust) are important factors. See Issue sanding between coats of polyurethane for more.
OR ... should I be using a different finish?
A smooth finish can be achieved faster, but with much more effort, using French polishing which is a traditional method where you rub on shellac. Once they see the (considerable) effort needed to apply shellac this way most people look elsewhere2.
Depending on the project you could try spray lacquer, which is a very fast-drying finish like shellac. But only small projects are really practical to finish using lacquer from spraycans, anything fairly big really require a spray setup (compressor and spraygun).
I'd like it to be sealed so if a drink or something is placed on it, the wood will be safe.
In terms of consumer-level finishes that are widely applicable to furniture work oil-based poly is about the best for waterproofing, in addition to providing the best scratch-resistance. It's also cheap, available everywhere and is easy to apply to a high standard. These are all reasons why finishing guru Bob Flexner, among others, recommends it so highly.
1 The first coat of most finishes cause a slight roughness to arise on wood that has been smoothed by sanding (can be seen less after scraping, and particularly after planing} although waterbased finishes are by far the worst offenders for this.
2 Plus it's also not very waterproof and easily damaged by any alcohol, which discounts it for many furniture projects in the modern home.
