Run tracert and then ping hop 2 and see if you get the spikes. This represent the segment between your house and the ISP.
Ping spikes are almost always a result of a overloaded connection. The actual fiber or even a ethernet cable has no place to store data so it is some device that delays it by keeping it a buffer. It in general does this because the data connection is busy sending other data.
The most simple case would be if you were download a large file at the same time you do the ping test. The download might force the ping packets to be stored in a buffer for a short time. In your case it is highly unlikely since the connection is large.
The other condition is the ISP sold the fiber to too many houses and the total bandwidth between all the houses is being exceeded. The ISP will seldom admit this type of problem.
Now if you find nothing in hop 2 you can try hop 3 etc but the father away from your house you get the less likely it is going to be you can get it fixed.