The most anyone can do with your IP is know the City, Country, ISP, etc, nothing more
They could also ddos (taking your internet offline)
You can always change your IP. If it's dynamic, it should change the next time your router/modem reboots, or just randomly. You can also call your provider...