I have extensive DxTory experince, but it would be nice if you could post up some video clips so I can assess what the problem is.
Are you closely monitoring the video capture frame rate? If it's not staying at a steady max of 30 FPS, or whatever you've set it to (should be at least 30), then it's not going to look as smooth.
Your settings should also ideally allow decent gameplay frame rates. Of course the bitrate you compress with afterward has a lot to do with the final look too, and I recommend doing that afterward, not on the fly. Hi res captures also require quite a bit of bitrate.
If your frame rates during capture are not consistent enough, you can boost capture performance significantly by using the Distribution Writing feature. To do this, you have to assign 2 or more drives to write the capture file, and set output to RawCap, instead of AVI. You also have to use the DxTory codec. Note that the capture file size will be huge if you don't set it to Low though. Then after capture, you use the built in RawCap Convert tool to convert them to AVI.
Lastly, unfortunately YouTube's flash conversion compression always makes videos look worse after upload, but there's a trick you can use to minimize that effect. When uploading 1080p vids, if you use VirtualDub to resize to 2048x1152 while running your compression, it tricks YouTube into using a higher bitrate than it would for 1080p, yet it will still be playable in 1080p.
Here's a handy tutorial on how to use DxTory.
[video="https://www.youtube.com/watch?v=W0KqsTa_HpM"][/video]