yes so i understand that if i use malicious programs in RDP, it has no way to harm my system (using random RDP on the web, not self hosted)
but if i run a VM on my own system, is it encapsulated enough to protect the host system from all the malicious programs in VM ?
"use malicious programs in RDP"
No, it does not work like that.
Remote Desktop Protocol is just the means where you can log onto a different system. It is not a "thing" to run applications.
Anything you run would live on that other system. You're just another user.
A VM can be OK, but some malware can escape the VM to the host system. Especially if you have shared resources. (Drives and folders)
Or, some malware detects that it is in a VM, and shuts itself off...