Interesting.
My suggestion is at this point to go online and look for some applicable videos (YouTube and otherwise) explaining and showing key and switch replacements.
For example:
https://www.youtube.com/watch?v=PqFa6EIwhRQ
Cannot attest per se to the video as a whole but starting around 2:00 or so he demonstrates removal of the cap and then the switch.
Then look for similar videos and compare. Two things to watch for: 1) that you are following the correct replacement procedure and, 2) that the necessary force does seem to be "normal".
And take a very close look at your switch removal tool. Appears okay (comparing your image and the video's image) but if the tool is not quite what it should be then the tool itself could be...