The PSUs in prebuilt systems are often very bad and barely adequeate to run the system as is! I wouldn't trust it to run such a power muncher, especially not through molex adapters. It's something about specific rails inside the power supply which can be overloaded.
You really need to consider replacing the power supply! Those RX5xxx series like power. I have the 590 which is essentially an overclocked 580 and it needs both a 6-pin and an 8-pin connector. The fact that there is no PCI-e connector on it could indicate it wasn't intended to power such a device.
Edit - I have had an FSP PSU myself. As far as I know they're okay, but the one i have is a 350 watt in an office PC.