YOu might have two different kinds of memory. A lot of older boards accepted EDO or SDRAM DIMMS (yes, beleive it or not, there were EDO 168-pin dimms). YOu cannot, however, run both at the same time.
If you have a configuration screen after POST (the kind that tells you what kind of drives your using, how much cache, etc.), and it tells you what kind of memory your running, look at it next time you boot. Then shut down, switch modules, and boot again. If your 32 says EDO, but your 64 says SDRAM, you found your problem.
Cast not thine pearls before the swine