As for the memory speed, the speed it will show during startup is the speed at which the RAM is currently running. The RAM clock is set as a ratio of the FSB clock. If your FSB is 266MHz and you're running a 1:1 ratio, your RAM will be running at 533MHz (266 x2 because it's DDR2).
You can select different ratios in the BIOS config, the setting is right below the FSB setting. Rather than displaying the ratios (1:1, 5:4, etc) it shows what the memory clocks will be at the available ratios. To display PC2-6400 at boot you'll need to run the memory at or near 800MHz. Most things I've read suggest the 975x chipset performs best with a 1:1 ratio. PC2-6400 RAM is a popular choice for C2D overclocking because it matches perfectly with a 400MHz FSB.
As to the random reboots, I'd suggest checking your temperatures with CoreTemp or the Intel Thermal Analysis Tool (TAT). Also, download Memtest86+ and let it run through its test at least 4-5 times to see if you have bad memory. If you get errors with both sticks installed, try the tests again with one stick and then with the other to identify the problem stick.