There are all kinds of logs in pfsense. I have not used it in a couple years so I forget exactly how you turn it on. It can create huge files that show all the firewall/nat rule hits. You should get a clue.
pfsense is a nice tool but it is complex so it is easy to get these rules just slightly wrong.
Worst case you can run tcpdump and capture the data to be really sure you are really receiving the udp data.
May not be the case but if you are trying to do IPSEC be careful IPSEC uses a different PROTOCOL not a different port.
As said above the log is your friend, it's enabled by default, under diagnostics. Be mindful of floating rules and where the forwarding rule is in the firewall stack they are processed from a top to bottom, by default all ports are closed, your port forward should be above your block any to all rule.