First transistor did not use silicon, but germanium and gold if I recall.
Integrated circuits are not that old though. Earliest examples were used in the space program in the mid to late 60s. Everything prior was made with discrete transistors and consumer electronics continued that trend for another decade. (See transistor radios for good examples of the latest in consumer electronics at the time)
Personal computer's were barely a thing in the early 80s. So these more recent integrated circuit process nodes have slowed down a lot in only 40 years.
But there are workarounds. You can already see it with AMD's 3D V-cache, and HBM memory stacking. Chiplets make production more efficient and universal interconnects will make having multiple chiplets have less of an impact when it comes to latency vs a monolithic design.
There are certainly already parts of the circuits that have gotten as small as they can be. Only the dense transistor areas are nearly as small as the advertised node numbers (which you can take with a huge grain of salt) They really need to adopt that LMC metric to make chips more comparable.