I'm currently in the process of renewing an old wooden floor. What would one use to fill in the gaps between the planks?
Nothing unless there's a draft problem. The gaps aren't generally an issue except visually but they can make for a cold floor depending on the rest of the structure of the house and how much air movement can take place through the floor.
Some argue such gaps are supposed to be there, and if you look at old floors 99 times out of 100 you'll see gaps between boards, sometimes quite wide ones, and in many cases there's no filler and no trace that there ever was any present.
In the past where gaps were filled they used various methods, including the same way gaps were dealt with on ship decks, using oakum and tar (!), as well as with melted wax or wax/resin compositions (also used as a filler material for furniture in this era).
See more discussion on the issue and solutions at these links:
How to Fill Gaps In a Wide-Plank Wood Floor on This Old House.
My home's pine flooring has large gaps in between some of the planks, especially in the kitchen. What can be done to fill or seal them? on Quora.
How to fill gaps in a wood floor? on HomeOwnersHub.
Filler for ancient wood floors? on the WoodenBoat forum.
I've asked in the local home repair store and they gave me a silicone like mass that resembles the color of the wood.
That seems to work ok for the nail holes
That stuff is no good for filling nail holes on a floor. This type of flexible filler is for filling gaps that require some movement to take place without cracking, as in gappy installations of skirting to walls, one wooden moulding piece to the next where some shrinkage and expansion will take place through the year.
If you must fill the nail holes (you don't have to, it's quite normal for them not to be filled) you want a rigid filler and ideally a strong one if it's to take direct foot traffic, i.e. if there's no carpet going on or any rugs being put down.