Could be a motherboard issue, a front panel switch or switch wiring issue, PSU issue, wake-on-something enabled in the BIOS that gets triggered as soon as the PSU gets power, the BIOS being configured to automatically turn the computer on, maybe more.
Start by checking the BIOS for a power option that has to do with power-on. It usually has three options: "On", "Off" and "Last state" or something along those lines, also check for any power-on-LAN/Mouse/Keyboard/etc. options and turn those off too. If your PC still powers up when you turn the UPS on, disconnect all non-essential internal and external components to rule those out. If the problem goes away, add them back in one at a time to find the problematic one.