Firefox clears old history automatically so it doesn't degrade performance. Enter about:config into your URL bar. First, use the searchbar at the top for the word transient. This is what the value is currently set to (mine defaults to 104858). This is read-only variable. If you want to set it higher, right click on any entry and go to New>Integer. Enter name of places.history.expiration.max_pages and set the value to something higher than your current value.
I'm not exactly sure what browser.history.maxStateObjectSize is related to, but you may need to increase it as well. It's set (by default on my PC) to 2 bytes (65536)
Maximum for either entry is probably 2147483647 (maximum value of an integer).