Can you confirm that the computer has a standard I/O panel? The boards I am seeing from the Nitro seem to have one. Should be a perfect rectangle that surrounds the rear I/O and is capable of being popped out. If it does, then I think you are right that a Mini-ITX board will fit.
The Acer board is proprietary, but it does appear to have standard Mini ITX mounting holes. What makes theirs special is that the front I/O is hard built into the motherboard. So any replacement motherboard would mean abandoning the front audio and usb ports. Possibly the power switch and reset switch would also be non-functional. So you will either have to rig something up to act as a power button or just short the standby pin to ground permanently like I mentioned previously, and then use the PSU switch to turn the system on and off. (That is all the power switch does by the way, short the standby to ground, so any low voltage (or high voltage actually) push button switch is enough.
A new chassis would avoid all that. You could then have standard front I/O/ power switch and all that.