Probably. The reason is because a lot of sandpapers, especially finer grits, are made with metals and fine particulates from the sandpaper could be left in the port, even if blown out with compressed air. Besides which, it's probably almost impossible to get a piece of sandpaper in an area that small in any way that could accomplish anything meaningful. Also, you could generate an electrostatic charge that might zap something on that circuit that the port is connected to. Doesn't have to be assembled for that to be a problem.
I'm sure there are ways to do what you need to do, I just don't know of any. Perhaps with sandpaper, a NON-metal based sandpaper that is, then a rinse with electrical contact cleaner, agitate with toothbrush, then rinse again with electrical contact cleaner, and then blow out using compressed air.
Generally, if there is significant enough corrosion or rust to warrant a need to do something, that something involves throwing the part in the trash.