You can get a 3.5mm extension cable, and have it on your desk so you just have to unplug the one and plug in the other.
There are a whole bunch of instructions to make one yourself online - it's a very simple build.
There are also 3.5mm audio switches like the one below, but they're a little pricy for such a simple thing.
http://www.amazon.com/Sescom-SES-IPOD-AB-Stereo-MP3-Player/dp/B005TE7GP4