It's Hexadecimal, actually. The PSP system is just telling you how the wireless passwords should be entered, and in this case yours did not work.
Please make sure the password is correct and check what kind of encryption your outer is using (WEP, WPA, WPA-2 PSK/TKIP). It should be the same on the PSP.