If you only need the video feed, you could just get a displayport switch. Monoprice has them for around $40 and amazon has a bunch ranging from $60 and up.
https://www.monoprice.com/product?p_id=24568
https://www.amazon.com/LINDY-Port-4K-DisplayPort-Switch/dp/B00ZK2S71W/ref=sr_1_3?ie=UTF8&qid=1544853672&sr=8-3&keywords=Displayport+switch
If you wanted to go with one set of keyboard/mice, you could look in to a Displayport KVM. There's a bunch of options but be warned the cheaper ones seem to only do 4k@30Hz instead of 4k@60Hz. Also KVMs tend to introduce a bit of input latency.