VGA or HDMI will not get you the kind of connection you're looking for. VGA and HDMI are for video, HDMI can also carry audio. However, neither are meant to transfer data like files and whatnot. The cable may be capable of this type of transfer, but the devices within the PS3 and PC will not accept an incoming data signal. This method will not yield a favorable result.
You'll want to plug both devices into a network device, like a switch or router. If the PC is configured to be discoverable on the network, the two should see each other and be able to exchange files. Please note that they both need their own displays. Just like when gaming, the PS3 needs its own screen.
Another way to do this would be to get a crossover cable and...