You are never going to have the compute resources to train a model like GPT-4.A trainable, offline model is the best way to do it, IMHO. I want the flexibility to work in highly specialized contexts.
However, what we could get is a model that can figure out what information it needs to look up, find it, and then reply with that new knowledge at its disposal.
This is also what's ultimately needed for Toms' HammerBot, I think. People expect it to have memorized every single detail about every model of every piece of hardware, but LLMs don't work like that - they're not databases. They pick up on general patterns and concepts. If a fact is repeated enough, they'll learn it, but it's not sufficient for them to see something only once - or even a few times.