Hmmm, well that makes sense in terms of your memory requirements.
You’re kind of stuck in a “catch 22” with your motherboards I think. Ideally it would be nice to have the best of both worlds, but you might need to compromise.
I think in your scenario based on what you’ve said your priority should be the memory and ensuring you have sufficient memory for your application to perform properly. On that basis you either need to do one of the following:
1) Find some old DDR2 that's suitable for a decent price – say 8 GB of it or whatever DIMM combination you need to bump your memory up to at least 6 GB, but I would give yourself some extra overhead and go for a minimum of 8 GB (6 GB for the application, let’s say half to a gigabyte for the OS requirements, leaving a bit up your sleeve free – hopefully). Maybe consider the second hand market (e-bay etc), or trying to source some cheaply from friends. Do you have an companies that sell refurbished computers in your town or region? If so, they can be a source of old memory. Or ring around shops and see if anyone has some old DDR2 that they will sell for cheap.
2) Cut over to the new Gigabyte board, get some DDR3 and put up with stock clock speeds. Not the best, but better than the current scenario – where your application is not able to perform properly.
3) Cut your losses and move on to another platform completely – selling the old one to recover some of your costs – might be an option, depends on your budget. I’ve been in similar situations to yours before (not with video applications though) and some times you need to bite the bullet and realise that the best solution is to do things properly and bear the costs of an upgrade.
4) Another idea, but I would only go down this path if you are “super cheap”, would be to leave things as is, rely on virtual memory usage for the application and see what you can do about seriously improving the speed of your virtual memory. RAID0 on SSDs would be a pretty good home solution and might bring your virtual memory performance up to the point where your application performs “ok” and produces decent results. I am mainly suggesting this idea if you have an SSD or two floating around or some really, really fast hard drives etc. But the previous ideas are much better solutions and this idea might still not give you the result you want, but if you had the drives lying around, it's pretty cheap to try out.
See what your performance monitor log file tells you in terms of your current memory use. But I think it will confirm that your system needs more memory. You might want to include some of the pagefile counters in your performance monitor.
Eg: here’s some of the basics - high level to begin with:
http://support.microsoft.com/kb/2267427
If you want more detailed info, I can easily provide you with a lot more good documentation from microsoft on this subject, as I've done a fair bit of work in the memory / performance space in my day job.
This counter is probably useful to you to confirm that your page file is being used and roughly how much.
Paging File, %pagefile in use