Why can't I apply the shellac at humidity greater than 85%
A few reasons.
One is that a very high water content in the air can slow drying, as you've experienced (a thin coat of shellac would normally take no longer than 10-15 minutes to dry) or prevent proper drying.
The main reason though is that as alcohol evaporates from a surface it cools it — just as you can feel if you get some on your fingers — and as the alcohol evaporates out of the shellac coat it can chill enough that it actually condenses moisture out of the air onto the surface. Shellac is moisture-sensitive as you probably know and this condensed moisture can be enough for the shellac to go cloudy. This clouding can sometimes be fixed, but obviously it's better to avoid it if possible.
But, as with most instructions on finish labels the 85% humidity thing should be taken as a general guideline rather than an absolute. I regularly apply shellac at humidities above 80% because I have no choice: it's always that humid here. The reason this can work is that relative humidity doesn't directly correlate to an absolute amount of water in the air, it's tied to the temperature. So it's much much more moist in the tropics at 100% humidity than it is somewhere more temperate. Also, how heavily the shellac is applied is a factor since the thicker a coat is the more alcohol is evaporating and the cooler the surface can get. So thin wiped-on coats can work without a hitch even when it's very humid.
Now that said, it is probably a good idea not to use shellac on the wettest of days. It's a pain, but a product's limitations are what they are and we have to work within them.
and if it was a big mistake how can I fix it?
It's only a mistake if it caused an issue. If your shellac eventually dried right — hard, not sticky and with no clouding — then you're fine.