You're supposed to convert the RawCap files first via DxTory's RawCapConv tool (purple tab bottom left corner of DxTory GUI). As far as I know, it's only designed to convert them to AVI. Then you'd need to use another 3rd party conversion tool if it's not compatible with the editor you're using.
If this is the process you followed, you probably had the output parameters of the tool set incorrectly.