I gouged out a rotted section on my outdoor deck rail surface approximtely 12" long & 2-3" deep. Advised to fill with sawdust/glue mixture. Should I fill all at once, or build-up/fill gradually...if so, how many inches at a time? Thanks so much!
I'd advise not trying to fill a void that big with glue and dust. Wood doesn't really like being treated like car bondo. Smaller voids and cracks can be filled with putty or home-made filler, but in general it will not last outdoors. Larger voids need to be filled in successive layers, and the layers will be more like 3/16ths of an inch or less. A 12" by 2" void is missing lumber, not a void to be filled with glue.
Either replace the member completely by scarfing a new length in, or chisel out the void and fill it with a "key" piece carefully fitted and glued to the void. Once cured you pare it to size with a chisel and/or plane.
It sounds like you need to replace the piece completely, but a photo added to your Question might answer that.
It really depends on how badly the piece is damaged, how it interfaces with other members, and how essential to load-bearing the affected member(s) are. If this is part of a deck that is there for safety and is intended to hold the entire mass of a stumbling human, then you absolutely cannot fill such a void with any kind of filler glue and junk. It is better to not have any safety railing (or whatever this is) than one that will give way immediately under a sudden load.
Not only is this contrary to any building code where such codes exist, it's the sort of thing that leads to broken bones and death.
Advised to fill with sawdust/glue mixture.
I'd use epoxy rather than woodworking glue. People glue boats together with epoxy. I've used it for some small deck repairs (much smaller than yours) and it worked fine. It'll be best if you cover it with something -- paint, stain, or UV-blocking varnish. UV light will break down the epoxy over time, and even if just the top part erodes a bit, that'll become a low spot that traps water, leading to more of the damage that you're trying to fix. Very fine sawdust works best; they even sell "wood flour" that's just very fine sawdust as an epoxy thickener.
Should I fill all at once, or build-up/fill gradually...if so, how many inches at a time?
I'd do the whole thing at once* tapping with a hammer as you fill to help bring air bubbles to the surface. Over-fill it just a bit, so that once the epoxy has cured you can sand it flush.
On the other hand, epoxy cures in an exothermic reaction, and if you mix up a lot of it all at once, it'll heat up fast and cure before you can put it where you want it. You didn't say how wide the void that you want to fill is, but if it's more than maybe 1/2", mixing in two batches might be a good idea. You might also consider using less of the sawdust in the first part, so that it's easier to pour and get down into the bottom of that void.
*Update: After a little more thought, I want to emphasize again the part about epoxy curing exothermically. If you have enough epoxy curing in a concentrated area it can get hot. Really freaking hot! Hot enough that Why is my epoxy smoking? is a pretty common question. Easily hot enough to melt a plastic mixing vessel and burn anyone holding it. And the hotter it gets, the faster it cures and gives off even more heat. None of that will happen if you read the directions for the product you choose and use some common sense; just don’t mix up a big batch to try to fill a large void all at once.