You don't have to worry if you don't want to. Windows will notice the old hardware has disappeared and not load the drivers for it, but they'll still be hanging around on your HD and in your registry.
There is a small chance however that a conflict could pop up between one of your old drivers, and a new one.
I'd just switch the hardware and let windows have a go, if everything works, no problem. Otherwise load up in safe mode and delete the hardware you removed.
I agree,
I have done this a couple of times. Just yank everything out replace it and windows takes care of the rest. Of course with windows you never know what could happen.