Some need a stronger push than others in my experience, especially if it's a new PC or a new power cable. You certainly shouldn't be able to accidentally knock it out.
With the other end unplugged from the mains socket (just as a precaution), give the PSU plug a firm push whilst pushing the front panel of your PC case in the opposite direction.