It depends on how many clients you'll need, what they will do, etc. You first need to determine if 4 GB is enough or not. I setup a Windows 2003 Server for a customer having over 20 users connected via RDP and it uses less than 3 GB. Knowing how much memory each instance of an application requires is important.
We use PCs to connect to RDP sessions (using mstsc), but you can buy thin clients. Check what HP (http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12454-12454-321959-338927-89307.html), Dell and others have to offer. Thin clients are nice, but you won't save much money over normal desktops. What re you trying to achieve? I mean we normally select thin clients and RDP to minimize the level of support and accommodate for remote users, not to necessarily save money on hardware.