This looks like a classic case of surface contamination to me, where the varnish has crawled away from something it can't stick to and settled at the edges of the contamination.
Although it doesn't usually happen on so large an area as this edge (often it's just tiny spots, the dreaded "fisheye defect") it looks a lot like the kind of defect you get if you accidentally get a product containing silicone on the wood. This is one of the primary reasons it comes so highly recommended not to use any silicone waxes, caulk or polishes* anywhere in the workshop.
Can't guarantee it'll work but one standard way to address this problem is to coat the problem areas with shellac first to seal off the contamination from the varnish. Once the shellac has dried (about an hour at room temperature will usually be enough, wait longer if it's colder) the varnish should go on normally.
*Many modern furniture polishes contain some silicone, one of numerous reasons they're a terrible way to care for wooden furniture.