You definitely do not want to decrease your hose size as you get closer to the dust collector/vacuum. Just about every shop I have worked in has had confusion over dust collection piping, usually because they try to think of it like an HVAC system, or something that is pushing air. This is wrong, because you are pulling air, so if you think about it, when you step down from a 4" to 2.5" you are only able to pull as much air as fits in that 2.5" hose back to the vac, so in the 4" hose that same volume of air will be moving MUCH slower from the tool until it gets to the 2.5" portion.
Inversely if start smaller, say 1.25", then air flow moves faster until it gets to the larger hose. Think of this like pinching, or partially capping the end of a hose. You will notice a difference in the pressure, and thus the velocity.
Usually, you want to avoid changing sizes except when branching in another line, and then you need to increase the trunk line to accomodate the added CFM's.
Here is a good source of information for designing a system: Dust Collection Design