Wow, that's a tough question. A patch panel is so that you have somewhere to terminate one of of cable runs end, so things can be moved easy. U can certainly connect, nicely, using a standard patch panel. A switch will be needed, for all practical purposes, and regardless of the size of your patch panel or number of desktops, for the network to finally function, yes.
If you are just the cabling contractor doing the wiring and not your responsibility to make the network function, then placing a switch in there is not in your purview and you can ignore.