I haven't pulled mine out yet, since you have the front panel off and the screws removed you may try wiggling it out the slot it's sticking through with the panel that has the phanteks name plate. If it's too tight of a fit that way, just remove the right side panel and with a couple of thumb screws the hard drive sleds come out. Then there should be all kinds of room to remove the fan and install the smaller ones.
You might need a screwdriver, despite being 'thumb' screws mine were pretty tight and wouldn't come loose without one. You'll need the right side panel off anyway to unplug the stock fan and route the power wire back through the case and grommets as you take it out. With the drive sleds (both) out of the way, you should be...