Ah! If you're talking processing speed, and I think you are, 2GB is essential, 4GB will be even better. This reduces disk swapping (virtual disk) Also, make sure your SD card has the fastest R/W speed.
The cluster idea is a good one, again the cost (including development time) vs minimizing processing time.
But if speed is really important, start with a PC motherboard with a really fast processor, add a boatload of RAM and an SSD drive. If you're doing automated processing, don't you don't need a high powered graphics card, If you are doing the tweaking, then a good graphics card is essential.
I'd go with a minimal Ubuntu. My application was automated so I used Imagemagick which is a very powerful command line software tool.