Chances are it won't work with a KVM at all. But it also depends on where your adapter is.
If I understand what you are saying, you are using it backwards. It is designed to convert an output to VGA, not take a VGA input and convert to display port.
Correct: GPU DP -> VGA Monitor
Incorrect: VGA display signal -> DP Monitor
So if you had a VGA output graphics card, it also wouldn't work.