It does matter - wood drill bits won't work on metal (destroying the bit in the process, unless used on thin and soft metals) and metal drill bits will increase splintering and tear-out when used on wood (but this depends also on the wood type and the diameter of the bit, and for smaller diameters there is little difference between wood and metal drills bits).
As for how to tell them apart - wood drill bits have two distinguishing features: a spar (also brad) and a lip. The spar helps in keeping the bit centered when starting the hole and prevents wandering of the bit on the surface of the wood. The lip acts like a chisel, slicing through the wood fibers in the circumference of the drilled hole, improving the quality of the finish. This is most important for through holes, where the lips helps reduce "blow-out" of the wood when exiting the wood on the other side. Wood bits are typical black and silvery in color, but other colors exist.

Metal drill bits, on the other hand, have just an angle at the tip. They often are fully black (plain HSS), golden (often titanium oxide finish) or coppery (often cobalt steel) in color:
