My question is: After I do this but before I plane the boards, should I leave them sit out in the open air (maybe a week or two) to reabsorb moisture from the air?
I think it's certainly advisable, and other than the wait there's no downside to doing this so you lose nothing but a bit of time if you take this extra step. And kudos for realising already it may take a couple of weeks, that just a day or two wouldn't be enough.
The wood should ideally be stickered during this stage to allow both faces of the boards easy access to air, and if there is concern they might warp put significant weight on top (centred on the stickering, or right over the top with sticks underneath to transfer the load down at the right spots) to help minimise this, possibly prevent it.
The reason I ask is I am not certain if I am over drying the boards or not
You probably are in some cases. This isn't fatal by any means but it can affect the working characteristics of wood, negatively, and as it also wastes power it may be worth slightly altering your approach and deliberately under-drying, letting the last of the drying take place at ambient temperature (again, stickered and weighted) after the oven has done the lion's share of the work.
But if you want to process the material as quickly as possible, the wood is currently turning out the way you want, and you don't mind having the oven running this long then continue. There is one alternative you might want to think about trying however....
Something else to try: boiling
This sounds counter-intuitive to say the least (read: seems crazy), boiling can't actually speed up the drying of wood, can it? But it can because of where the water is held in green wood and what happens to some of it when boiled1.
The hard part of this is usually rigging up something that will allow the boiling of pieces of wood of useful dimensions, but as you're currently processing wood of fairly modest size this will be easy enough for you, and boiling is apparently really good at producing usable wood directly from green in a reasonable timeframe2 without a great expenditure in energy.
More on this in the links below:
Don’t Crack that Bowl, Make Wood Soup Instead (PDF)
drying wood using boiling method...data and charts
1 This is related to why you can't soak previously dried wood and get it as wet as it used to be when green.
2 It should be expected that the smaller the pieces the faster you can use them, with very small pieces you might be able to start working the wood as soon as it's cool enough to handle ^_^
