There are all kinds of logs in pfsense. I have not used it in a couple years so I forget exactly how you turn it on. It can create huge files that show all the firewall/nat rule hits. You should get a clue.
pfsense is a nice tool but it is complex so it is easy to get these rules just slightly wrong.
Worst case you can run tcpdump and capture the data to be really sure you are really receiving the udp data.
May not be the case but if you are trying to do IPSEC be careful IPSEC uses a different PROTOCOL not a different port.