Sounds like a loose connection, or bad speakers.
Try plugging some headphones into the same plug. If the whine goes away in the headphones, its the speakers.
Plug the speakers into something else, if there is no whine there either, than something outside your case is interfering with your speakers. Try a different plug on your motherboard . If the speakers have a power cord, make sure its properly grounded.