Not the best option but generally if you who have full access to the network can not access the router then someone from outside can not either.
To some extent you have to look at the actual threat. Most home user "attacks" are the neighbor kid whose parents have blocked their porn surfing so they try to use the neighbors wifi. They really don't care about attacking your machines they just want to use your internet. The wifi passwords generally prevent that.
They would have to first get past the wifi passwords. If they did that then your network is already unsafe even if you used good passwords. Its not like they can take the admin password and somehow get more access they already have full access.
This would more be a worry if you had someone you trust on the network that then used it to give themselves more access. BUT if you can not even use the ip because for example it is in the wrong subnet then it makes it even more unlikely.